Why you generate no insights?
Or the daily pains of an analysts.
Disclaimer: it’s heavy on the memes, which might sound like a big rant. But it’s a collection of my past experiences.
Being a data analyst is a challenging task. Most Data People juggle multiple responsibilities, sometimes stuck in meetings all day and dealing with shifting priorities. The mentioned things are only several of quite clear issues, why Data Analysts are sometimes not focusing on insights.
Uncle Data is teaming up with Turing College! They say they’re the #1 place to learn online for working professionals. Check out their Data Analytics or Data Science and Analytics Masters Degree programs to enter the Data Field or brush up your skills!
While Turing College will provide you with the necessary skills, Uncle Data will help you navigate the caveats of the industry and things that are not visible straight from the trenches.
Let’s talk about others and start with poor Software Engineering. Usually, Software Engineers don’t care much about data structure for analytics. Nowadays, they prefer even freaking JSONs in their RDBMS because why think before about the data structures when we can add/remove things as we go. I understand that it makes iterations faster, you can more quickly develop your product, etc. But why not to later make it better? Or maybe, from the start, use NoSQL DB? Or perhaps talk with analysts and agree on the formats and where to shove the data for a better analysis format? Anyway, this is a bit of a rant, but it still makes Data Folks jump through hoops to extract it;
We also have some poor code standards that cause different names across tables for the same values, etc. So if you’re a single analyst in the company, you’ll spend the majority of the time trying to understand the data and create tons of custom SQL and then when Engineers change schemas, you’ll have to do it over and over again.
Here comes the data contracts topic that at least was very hyped up at the beginning of the year but is now a bit calmed down. Does all the companies have it solved by now???
Data Infrastructure and Data Warehouse are next on my list. If infra is not in the right state, your queries will run longer than they should; you’ll join a bazillion tables to get some aggregation. Technically, it’s a job for a data engineer, and if they’re missing in the company or doing a poor job, you will be on the receiving end. On the other hand, even if infra is in place and DWH is there, how quickly will (and when!) your Data Engineers or Analytics Engineers cover your needs? What about the interface for you to access the data? If it’s UX is terrible or you’re missing some critical features, you’re also screwed.
Let’s also talk and give some attention to Data and Analytics Engineers. They’re in the same boat, dealing with fires everywhere, especially when the focus on data is so high. It’s very easy to leave small mistakes, typos, and a bit different metrics definitions just because of tons of context switching and ever-changing priorities.
Tooling for proper metric managing is not there yet; data quality has just started to be the focus, and for the folks starting in Data/Analytics Engineering, some of the tools are too confusing or over-engineered - this leads to inconsistencies, firefighting and ineffective DWH structures, that leads to the Analysts pains.
So the things mentioned above force analysts to do hacks, but it works, so you’d say it's not big of a deal, right? But I guess what kills the buzz - is the measurement of success for insights. How do you know if your insight is good and not some generally available knowledge, just for some reason you were excluded? I.e. there is a spike in one marketing channel, you dig deep, and your traffic comes in from Google ads. There was a switch in Marketing campaigns, but you’re unaware. Here comes communication, which is always an issue in all company sizes, but the question of what’s a good insight still stands. It can’t be quantifiable easily, and it’s not like an engineering task, where the result is clear, tests pass, and your change is merged to the main branch. I've covered this in another blog post if we’re talking about ROI on analytics in general. Usually, data is just a cost centre because it’s hard to measure or think of a good and valuable metric to make it revenue-generating.
Let’s talk about data literacy. People will focus on shiny things - i.e. Dashboards.
I'm not saying it’s terrible, but sometimes they will be just used as an excuse: “We have dashboards, we’re data-driven”, even though the whole dashboard is based on vanity metrics that don’t show how things are going. This makes sense when you consider why analysts focus a lot on visuals and dashboards - they’re optimising for success. People will remember the colourful and pretty dashboard, not when you tell them there are issues in some segment. Here is another edge of data literacy. People tend to think that they know things better with their gut feeling. There is no data to back my decisions, but I’ve been in the industry for X years and know what the market wants.
Imagine that you worked your butt off for some analysis, gathered all the evidence, even prepared an action plan, came to the Product Owner/Engineering Lead and presented it, and if it goes against his gut feeling - you’ll get a reaction you’d expect.
And last but not least - Data Strategy. Strategy, to me, feels very tied up with big consulting companies like McKinsey, etc., but since it’s Data and I have some thoughts about it, I’ll let this one slide. If you just now hear these words in your company, management most likely found some articles from consulting companies that sound like, “If you want a successful business, you have to have a data strategy”.
Depending on who’s running the data strategy, you should be able to determine if it’s hype-driven or well-thought-out. People leading this should, in my eyes, come from the trenches. You might ask why. Because they know what hurts and what’s missing, and they should be able to force upper management to make needed changes in the company. Otherwise, it’s just for show.
Another reason is that the focus areas will be relevant to the current setting, so it won’t be similar to “Let’s do Machine Learning” for the sake of doing ML. There will be a purpose. Keep in mind that a good data strategy involves not only technical things but also education. As I’ve written above, data literacy is essential, and just because it’s a habit, it will take a long time to create such. We, as humans, are complex beings, and adopting new things takes time. As for the people pushing for changes, you’d need persistence and resilience since results won’t come overnight.
Dear Data Analysts - stay strong and push through for a better tomorrow! We all Data Folks feel similarly in a way!