Understanding the Role of a Data Engineer

In this article, I will outline the key responsibilities of a data engineer, based on my experiences at a large bank where my team focuses on deploying machine learning models. Having worked in this position for about two and a half years, I've begun to understand the intricacies of the data engineering field. This analysis aims to provide a comprehensive overview of my work activities, although it is based on approximations rather than minute-by-minute tracking.

Daily Task Management

To stay organized, I maintain a list of tasks in a notes application. From October 2020 to December 2021, I compiled these tasks and categorized them for analysis. While I was unable to track the exact time spent on each task, I believe this categorization will shed light on the duties commonly undertaken by data engineers.

Insights from Task Frequency

The analysis highlighted the most common tasks I faced during my first year as a data engineer, with learning and coding topping the list.

Categorization of Responsibilities

To better understand my responsibilities, I organized them into several categories: learning, coding, troubleshooting (including code, pipeline, and integration issues), meetings, administrative tasks (like emails and time submissions), documentation, compliance (involving JIRA and mandatory training), infrastructure requests, and visualization (such as dashboard creation).

Here’s a glimpse of some raw task data.

The tasks of coding, learning, and troubleshooting are quite apparent and align with the expectations for a data engineering role. However, I was surprised to find that a significant portion of my time was also dedicated to administrative duties, documentation, compliance, and infrastructure requests. Given the nature of my employer, these compliance tasks were both frequent and demanding. I often found myself grappling with infrastructure requests due to the complexity of the systems in place. Nonetheless, this experience has proven valuable, and I have noticed improvement over time.

In terms of my primary responsibilities, learning and coding involved various activities such as training sessions, online research, writing code for pipelines and tools using bash scripts, Python, and SQL, and performing code tests. While it was unusual to write SQL from scratch, many templates and resources were readily available. It's unclear if the high frequency of learning tasks stemmed from my novice status or factors related to our tools or the nature of machine learning operations. However, given my team's ongoing development of pipelines and tools, it makes sense that coding emerged as a prominent task.

My Journey as a Data Engineer

Initially, I did not set out to become a data engineer, but I embraced the opportunity when it arose. Over time, I’ve discovered that data engineering encompasses various specializations, much like other tech roles. The specific duties of a data engineer can vary greatly depending on the company and its infrastructure. Larger organizations often offer more avenues for specialization but may provide fewer opportunities for innovation.

I feel fortunate to work within a team that focuses on operationalizing machine learning models. Collaborating with data scientists makes my job fulfilling, as our efforts have a direct impact on business outcomes.

While I’m not boasting, I genuinely find being a data engineer to be an exciting role. There is a noteworthy intersection with data science, and the specifics of the job largely depend on the workplace. Looking ahead, I aim to further my expertise in machine learning operations and deepen my understanding of databases, algorithms, and cloud infrastructures.

Thank you for taking the time to read this! If you are a data engineer, I’d love to hear your thoughts on whether you find this analysis of our typical tasks to be accurate!

Chapter 2: A Day in the Life of a Data Engineer

This video offers a glimpse into the daily routine of a data engineer, showcasing the various tasks and challenges encountered throughout the day.

Chapter 3: The Role of a Data Engineer in Business Context

This video explains the significance of data engineers within an organization, detailing their contributions to business operations and decision-making processes.