I automate things so that people don't have to keep track of literally thousands of spreadsheets in order to measure business metrics I'm on a team of people who makes buildings out of Legos. There are the gatherers, who gather Legos fro...

The full context of analytics engineering is to be found in The Analytics Engineering Guide. The Analyst, The Engineer, The Visualizer Since the problem space is so varied, we align our analytics professionals with the listed business area ...

Well, notebook programming environments can be really handy for doing certain types of interactive or exploratory development. There is quite a few of them. SQL & Python Deepnote Free for individuals/small teams SQL + Python ...

Users of a silly language called SQL Indirectly described by Jamie Brandon in his Against SQL The relational model is great but SQL is the only widely-used implementation of the relational model, and it is inexpressive, incompressible, and non-porous. ...

You will always need that query again Queries are living artifacts that change over time If it’s useful to you, it’s useful to others (and vice versa)

… but plan for tidying it up! Data isn’t objective, and analysis isn’t structured. It’s just as creative—and just as messy—as the papers I ran away from in college. But we can’t define metrics informally; our instructions have to be precise. And fo...

If you have DEs working on your analytics stack, you either don’t have a Modern Data Stack or don’t actually have DEs.

Pick a data job that fits you Go from a good to a great data analyst

Problem Headless BI Imagine instead that you could disentangle metric definition from visualization. In this world, the teams that own metrics would be able to define them once, in a way that’s consistent across dashboards, automation tools, sales report...

OpenRefine is a powerful free, open source tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data. Web Scraper makes web data extraction easy and accessible for eve...

New job? New data set? First time using SQL? First time seeing that file? Reviewing someone else’s code/output? SELECT * FROM table LIMIT 10 Highly Successful Data People have a habit: the instinct to first grasp, even in a shallow way, what the data loo...

Everyone is a data analyst, whether they know it or not and whether they like it or not. Training for data analysis is essential, but there is limited bandwith. We can automate or teach. None of them is perfect. Tukey JW, The Future of Data Analysis: ...

A huge amount of effort is spent cleaning data to get it ready for analysis, but there has been little research on how to make data cleaning as easy and effective as possible. This paper tackles a small, but important, component of data cleaning: data tidyi...

If there is a way to describe data analysis workflow in a diagram, this would be it: This fits within a broader PPDAC cycle: Problem: Identify the issue or question you want to address. Define the objectives of your study. Plan: Decide on how to appr...

A mature data organization, first and foremost, is a mature analytics organization. There are three tiers of data analysis: reporting (= gathering facts), insights (= understanding relationships between facts), and prediction (= predict performance throug...

Beware of "actionable insights": data are complex and you need to invest energy to understand them. It's not so straight-forward.

List of key terms and metrics with definitions; a business glossary Steps: Gather terms from different departments Give the terms a definition Find alignment Get support and sign off Centralize the document Upkeep ...

What is fizzbuzz? Fizzbuzz for SQL Implementation for free Paid implementation

Summary Understand the purpose of data analytics and make sure it is being met Always focus on data quality, without clean data any investment into reporting/analytics is wasted money Hire people who understand the data and avoid thinking biases. You...

The talk page is available here. More about the blue tape list. The whole presentation is here. Quit-hit vs Data-request Data Quarterly Feedback Form Analytics Framework My notes

Excel mistakes cost millions of dollars Some people fix them as a job Bespoke software simply isn't an option for some of the African clients. "As a result spreadsheets almost become the default – especially within governments." ...

PBCs charts, i.e. XmR charts → understanding variation → process control worldview → acting, not just analysing → knowledge I have never, not once, seen a data tool vendor or data publication explain how one might go from topics like ‘the modern data s...

Questions we ask in the data discovery process Where can I find data about…? What is the data about? Who can I ask for access? How is the data created? Can I trust it? How should I use the data? How frequently does the data refresh? Features ...

To increase data literacy across an entire organization, you need something often referred to as data democratization. That is, a data platform that works equally well for the marketing, sales and editorial teams. If a platform isn't architected to br...

By conceptualizing self-serve BI as a simplified means for doing an analyst’s job, we’re not only making the self-serve problem too hard—we’re solving the wrong thing altogether. When you ask an analyst a question, their first thought is often, “how might...

Next time you get asked to justify the ROI for your team, don't. Sit back, and let others tell the story for you. If your partners aren’t willing to go to bat for you like this, then it’s time to take a step back to rethink how you’re operating. ...

Incidents are a learning opportunity An incident postmortem brings teams together to take a deeper look at an incident and figure out what happened, why it happened, how the team responded, and what can be done to prevent repeat incidents and improve futu...

General idea Run data team as if you were building a Data Product that helps guide decisions and all of your colleagues are your customers. So what is the Data Product? It is the collection of every piece of data, and the tools used to generate, access...

"Analytics organizations don’t exist to write SQL. We are here to problem solve." Problem data-driven = we make our decisions using data -> it doesn’t recognize the hard work of data scientists/analysts as the focus is on th...

Learn CLI Alfred and its workflows: app for macOS which boosts your efficiency with hotkeys, keywords, text expansion and more Raycast: blazingly fast, totally extendable launcher IFTTT: integrate apps, devices, and services to work together in new an...

The Y2038 problem, often referred to as Y2K38, arises because many computer systems store time as the number of seconds since "UNIX epoch" – 00:00:00 UTC on January 1, 1970, using a 32-bit integer. This format can only represent time up to 03:14:0...

SELECT * FROM pony WHERE id NOT IN (1, 2, NULL) -- 0 rows, major wtf It's because: SELECT * FROM pony WHERE NOT ( id = 1 OR id = 2 OR id = NULL ) And then remove the parenthesis using De Morgan’s laws: SELECT * FROM pony WHERE ...

Overall Pull request size It should be small. The pull request must have a maximum of 250 lines of change. 10 lines of code = 10 issues. 500 lines of code = "looks fine." – Code reviews. Feature breaking Whenever its possible break pull requ...

It starts by defaulting to public for everything. Our code, issues, and documentation are public. The actual code that you use for analytics isn't your company's competitive advantage. The real value is the data itself and the decisions people m...

General Packaging Python Projects How to Package Your Python Code Creating a pip installable package Understand semantic versioning Choose the right open-source licence Github Create your custom Python package that you can pip-install from y...

Python GeoPy Geocoding API from Google Maps with $200 monthly quota JavaScript Mapy.cz

Who creates emojis When we started looking into who could approve our proposal for a dumpling emoji, we were surprised to learn that the emojis – so central to the lives of ordinary Internet users across the globe – was controlled by a handful of multina...

Why, really? Become shell literate: Attempt at trying to persuade you that this whole thing makes sense. Full disclosure: the author of the article is a well-known free software advocate, so he is far from impartial in his article. That said, he is certa...

How does it work? It learns by analysing large amounts of text data using deep learning algorithms and NLP techniques. This allows it to recognise patterns and make accurate predictions about the most likely next word or phrase in a sentence, which helps...

Intro The Missing Semester: What is the difference between Docker and a Virtual Machine? Docker is based on a more general concept called containers. The main difference between containers and virtual machines is that virtual machines w...

Intro It is a standardised way of tracking your code and analyses (plus history thereof). Mrshu says that it's a good idea to learn Git, because it: Helps avoid "versioning hell" (you know, files like essay.doc, essay_v2.doc, e...

Change your Node version with n (based on this) npm install -g n # install n globally n 10.16.0 # install particular version n lts # install latest version n # switch versions

Links Quora Discussion Automate the Boring Stuff with Python or book – a practical programming course for office workers, academics, and administrators who want to improve their productivity Learn Python The Hard Way, 3rd Edition.pdf 81 Python Code...

Intro First and foremost, consider whether to learn it all Dataviz Shiny Dashboard TidyTuesday by R4DS Online Learning Community Submissions by Ijeamaka Anyene. How to use R with Tableau (and when you should) ...

Learning SQL Climber SQL Zoo DataCamp Lost at SQL Window Functions – intro, questions Reading Keep in mind that relational database is just one of several database paradigms, even though one of the most commonly used. Learn why SQL should ...

Philosophy Optimise for reading code/text, not writing Be programmable by itself (keystrokes are composable commands) Avoid the mouse (too slow) or even the arrow keys (too much hand movement) In practice this means using different "modes of o...

YAML Multiline Find the right syntax for your YAML multiline strings YAML anchors Be DRY in YAMLs by using anchors

Test API through Postman or its open-source alternative Bruno

Entropy Entropy is a measure of randomness Hashing functions A cryptographic hash function maps data of arbitrary size to a fixed size An example of a hash function is SHA-1, which is used in Git references At a high level, a hash function can...

Blogs Martin Fowler

Software is a tool to help accomplish something for people – many programmers never understood that. Keep your eyes on the delivered value, and don't over-focus on the specifics of the tools. – John Carmack General Tools carbon Create an...

General CommonMark GitHub Markdown, incl. things like alerts Snippets How to reference local files Obsidian opens the file if it can, otherwise it opens in default app Obsidian, Syntax Same folder: Read more [here](./ano...

Intro Regular Expressions = "regex" or "regexp" Comes from the ed editor but you'll mostly encounter the grep program A quick way of describing a particular pattern of characters in text Allows for extremely effective search...

One-line summary A source-control branching model, where developers collaborate on code in a single branch called ‘trunk’1, resist any pressure to create other long-lived development branches by employing documented techniques. They therefore avoid merge he...

What are dotfiles? On your computer, there are hidden files called dotfiles, they’re hidden because they start with a '.' e.g. ~/.bashrc . they usually control configuration of tools or the terminal (command line) itself. How should you organise yo...

Building on open-source Advantage over Software Simply put, open source is a superior approach at building and distributing software because it provides important guaranties around how software can be discovered, tried, operated, collaborated on and packag...

In semantic versioning, a version is composed of three numbers: Major.Minor.Patch. For dbt, for instance: Major: dbt is going to look, feel, and work like this for a long time. You can know with confidence that, if the code in your project work...

Now let's talk about unnesting. The UNNEST function takes an array and returns a value table of the array's element type. Whereas most BigQuery tables are SQL tables defined as a collection of columns, a value table has rows of some value type. Fo...

All modern analytical database optimizers appear to treat our “import statement” CTEs as pass-throughs. These CTEs have no impact on performance whatsoever, and just act in the way that we want: as a great tool to clean up our code but not ultimately to cha...

When executing, the order actually is: FROM JOIN ON WHERE GROUP BY HAVING SELECT -- including window functions ORDER BY LIMIT

It's mostly believed that INNER JOINs over WHERE IN subqueries, i.e.: Not this: select count(*) from t1 where col in (select col from t2) But rather this: select count(*) from t1 inner join t2 on t1.col = t2.col Often the results from a corr...

I first encountered this function when trying to join two tables together using about eight separate fields. Not ideal. The natural inclination is to create your own ID by simply concatenating a bunch of fields together. These columns are bad because they...

Partitioning divides your table into parts and keeps the related data together based on column values such as date, country, region, etc. Partitions act as virtual columns. You define them at table creation, and they can help reduce the amount of data scann...

Intro From The Missing Semester: You are probably already familiar with the notion of daemons, even if the word seems new. Most computers have a series of processes that are always running in the background rather than waiting for a user to la...

Roles of a linting tool Style Nobody likes to nitpick on style in PR reviews, and nobody likes to get these reviews either. Not only is it a bad experience for everyone involved, it also gets in the way of the important conversation around business logic...

What I like to think of virtual environments as package bookshelves for each of my projects. If I’m working on a cooking project, there is no need for me to have a book on surfing. Creates special isolated environments where all the packages and versio...

A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away. – Antoine de Saint-Exupéry Links Context Modern Data Stack. Read what the modern data stack is and how came around. Find ...

Airbyte: open-source, big hit on GitHub, sustainable business model, setup with docker The downsides of full containerization are the inherent performance costs and the added steps when modifying or building new connectors. The Airbyte team seems...

OpenLineage OpenLineage is an open platform for collection and analysis of data lineage. site, Github, article Marquez Marquez is an open source metadata service for the collection, aggregation, and visualization of a d...

We saw a tremendous amount of innovation immediately following the launch of Redshift in 2012, unlocking brand new levels of performance, efficiencies, and new behaviors. We then saw a maturation period as these nascent products were deployed by the market,...

Star Schema & One Big Table Star Schema A star schema uses a central table, known as a fact table, to store measured or transactional data. A fact table uses aggregated facts to store key business information, such as financial data or logged perfor...

Don't use the same server as your production system Upgrade to pg 12+ (or avoid common table expressions in your queries) Go easy on indexes – less is more Consider partitioning long tables Ensure you're not I/O-bound Vacuum analyze after ...

Data infra: enables data-driven decision making (analytic systems) and drive data-powered products, including with machine learning (operational systems). Data infrastructure is undergoing rapid, fundamental changes at an architectural level. Buildin...

Trying to teach the habit of digging deeper Bullshit = when you create non-sense bc you don't care Calling bullshit = even on lies, injustice Clean the information bullshit around us, similarly to physical environment We all create bullshit = we...

  Expectations Reality   What does the data look like? Static dataset/table/CSV that represents observations or an experiment Constantly changing (new records, changing schema) with dozens (if not hundreds...

"Why do I need to learn this?" Starting with why can give employees the motivation they need on their upskilling or reskilling journeys. Use case studies and success stories relevant to employees by role to show them the benefits of data skills, s...

Different types, different context – the ones on which they can see themselves, that's why there is not a right answer Making just about hard Giving them stat nuggest of statistics info afterwards They work together – a teacher and a journalist ...

Understanding how it is taught. Interview with teachers. There is no formal plan, more used as a tool to explain other complex topics. Use a lot of pictograms. Something you can directly count, tangible objects. It's important to explain that there ...

Test They think that if they live with it, they understand it. Data fluency Easiest thing is to put it on the test Great mismatch between high school teachers (all maths) and college (core skills, rest we can teach): arithmetics (fractions), data a...

First, students must be educated on why data is important. It could be for employment, but we need to realise that education isn't here just to prepare kids for work. Think analytically and critically about the data that is presented to them, as wel...

The focus of data people is often to improve their or other people's knowledge of technical tools. It's perhaps because it's more tangible and easier to grasp. You go through a course and feel like you achieved something, expanded your knowled...

Assessment A Principled Way of Assessing Visualization Literacy VLAT: Visualization Literacy Assessment Test Certifications & Qualifications from Qlik Research Better the Data You Know C'est la vis: Visualization Literacy at Elemen...

General Manifesto for Data Practices Zbierka úloh na pohovory Analytických pozícii Descriptive, Predictive, and Prescriptive Analytics Explained Data The School of Data The School of Data aims to make your learning experience as tailo...

Data Literacy = f(Numeracy, Graphicacy) Numeracy = maths, stats, logic Graphicacy = graphs, charts, maps (explore or communicate) Technical skills, Communication skills, Domain acumen Combining analytical and creative brain hemispheres t...

Five years ago, Jack Welch famously stated that there are three fundamental ways to motivate employees: financial rewards, recognition, and a clear mission. A fourth motivational principle must be added: personal growth and development. Ideas: You can ...

Based on Deahl (MIT) The definitions: Understanding Data: Data literacy requires an understanding of what data is, what types of data exist, and how data is generated. A data literate individual must also have knowledge of the role and impact of data ...

What to do? Removing context in a way that what is true in some context is a manipulation in another. How to go against? "How can I disprove it?" Psychology (system 1/2 thinking, beliefs, too good to be true, biases: Anchors, Framing) Visua...

Foster critical thinking This means questioning your own biases, distrusting your intuition, and displaying a healthy degree of skepticism when presented with ideas and suggestions from others Invest in training So, the primary investment is not m...

Spotify doesn't actually use a single revolutionary recommendation model. Instead, they mix together some of the best strategies used by other services to create their own uniquely powerful discovery engine. To create Discover Weekly, there are three m...

General StatQuest with Josh Starmer – a YouTube channel on Statistics, Machine Learning, and Data Science. Statistics Seeing Theory – Statistics through Visuals – a visual introduction to probability and statistics. Fundamentals Of Statistics For ...

The main takeaway I will share is that companies consider three tracks of data science work to meet the needs of your business – Analytics, Inference, and Algorithms. The Analytics track is ideal for those who are skilled at asking a great question, expl...

Seduced by their seeming precision and objectivity, we can feel betrayed when the numbers fail to capture the unruliness of reality. Goodhart's law Once a useful number becomes a measure of success, it ceases to be a useful number. Wh...

World Dataset Search Dataset Search is a search engine for data sets. Using a simple keyword search, users can discover data sets hosted in thousands of repositories across the web. UK's Office for National Statistics ...

Mind that data doesn't perfectly represent reality, it's always only its biased reflection "Data is always translated to what they might be presented. The images, lists, graphs, and maps that represent those data are all interpretations, and ...

Non-profits in the social service sector often have a wealth of data at their fingertips, including data tied directly to the services they provide (aka program-level data), financial accounting data, and donor data. However, non-profit staff, who often j...

Economist don't know how to viz results of their work Viz is how to get your research out there, it doesn't just need to speak for itself People are ready for scatterplots Scientists think they are not able to think visually enough and need he...

Using language with a racial equity awareness Ordering data labels in a purposeful way Considering the missing groups Using colors with a racial equity awareness Using icons and shapes with a racial equity awareness Demonstrating empathy Questio...

Download the version of WGS84 The encoding is Czech, i.e. Windows-1250 (or sometimes Latin II) To change it for some programmes, it would work to change the *.cpg file from 1250 to UTF-8 For Tableau, you need to change the encoding of *.dbf file, idea...

Clear title – Does your viz have a clear title that draws the attention of your audience and sets the scene for the information presented? Include sources – Have you included the data source(s) and your details (name, Twitter handle, etc.) Remove clutte...

Check these to understand the data Check these before publishing Checklist by Ann K. Emery Checklist by Stephanie Evergreen

Identify the Challenges – Start by reading the article that we link to. Read about the topic. Write down keywords, definitions, acronyms, anything you can think of about the topic. Literally write it down on paper. Examine the Metadata – What are the data...

Links David Rumsey Map Collection Five charts that changed the world W.E.B Dubois's Data Portraits and an article Pictures

General Tableau Public Makeover Monday Workout Wednesday A weekly challenge to re-create data-driven visualizations Tableau Community Courses From Tableau Learning Site Training Resources Udemy Tableau 10 Basic on U...

Learning Learn D3.js in 5 minutes How to learn D3.js Scrimba: Course on D3 D3 Tips and Tricks v6.x CSS Basics Inspiration Visual Cinnamon: My Journey into Data Visualization Hesham Eissa: How to learn D3.js with no coding experience Obse...

"A data visualization hides as much as it shows." – Alberto Cairo "Viz what you love" – Zach Bowders Blogs The Information Lab Blog by several people learning dataviz The Data School Blog of seve...

Extract GPS points from shapefiles through Python Import to Sheets Connect and use as polygon in Tableau, best to blend data (and make points the primary dataset)

Apache Superset Apache Superset is a modern data exploration and visualization platform D3.js D3 is a JavaScript library for visualizing data with HTML, SVG, and CSS. Flourish Studio Flourish is a powerfu...

Theory Material Design Material is a design system created by Google to help teams build high-quality digital experiences for Android, iOS, Flutter, and the web. Charts Diagrammm Diagrammm has been developed as an onlin...

Data Stories: Seeing data We should open up dataviz for non-experts Subject, source/location (where they read it), opinion, time, confidence (language skills, statistics, computer skills, critical thinking), emotions (the above-mentioned context-based) ...

What is Makeover Monday? Makeover Monday is your weekly learning and development appointment with yourself and hundreds of passionate data people. For free For more information, check its official page or watch the video below. What to do? Go to ...

Be more inclusive in data viz, bc we are where art and science meets We need artists to better understand science and scientists to better understand arts The end goal is to create "Actionable wisdom" for everyone

“What you measure, you improve.” But how does this actually work? To understand effective, motivational data design, you need to understand the psychology of feedback. Effective dashboards provide effective feedback and rely on two dynamics: ...