Lokesh Learning Journals

Data has always been dubbed as the "new oil". A very fitting anology. Just as with oil, data is useless  and even dangerous in its raw form.  Just like with oil, which needs to  be refined, transported safely, and delivered to the right place at the right time in order to useful, data must go through the similar process. This is where the Data Engineer comes in. While Data Scientists are like chefs who create a masterpiece meal (the insights and AI models), Data Engineers are the architects and contractors who build the industrial kitchen. They ensure the water lines are pressurised, the electricity is stable, and the ingredients arrive fresh and sorted every morning. In this article, we will have a quick look into:  What is data engineering? what are data pipelines? what is ETL and ELT? why is data quality important? etc. 1. The Core Mission: Building the Pipes The primary responsibility of a data engineer is to build the infrastructure and reliability required f...

Docker is a open-source platform that has revolutionized how we build, ship, and run applications by introducing a standardised way to package software.  In this article, we will look into:  what is a container? How are docker containers and images different? How are containers and virtual machines different? And deep dive into Docker. 1. What is a Container? At its core, a container is a way to package an application along with all its necessary dependencies, libraries, and configurations. It serves as a portable artifact that can be easily shared and moved across different environments—from a developer's laptop to a testing server and finally to production—without any changes in behavior. Where do containers live? Containers are stored in repositories . DockerHub : The primary public repository for sharing Docker images. Private Repositories : Most companies maintain their own private registries to secure proprietary code. How containers Improve the Development Process?...

In the past few days, I had not been at my best. Not been at my 1% improvement everyday. I did not do the things that I said I will do everyday nor come to blog. I did get started on things and tried to study. That was good effort. But, I was distracted.  I have been distracted. It is not being a working adult. It is not easy to adult. It is different from when we were at school or university. We would just think, why not relax. You are getting paid with money, just relax. No harm, no foul.  In university, your timetable was broken down for you, you had semesters. You had assignments. You had exams. It is not the same in the outside, no one has set timetable for you. No one is actually invested in your learning or development. You have to manage it all.  It is easy to think that it is fine to relax. You are still getting paid. But, know this, life and its duties will catch up with you.  Time wasted is time wasted. It is the most important resource that we should safe...

Artefact Repositories  and  Artefact Repository Managers  are critical in the Continuous Integration and Continuous Deployment (CI/CD) of devops. They ensure that we can  store and manage the outputs of our build processes with speed and reliability.  In this article, we will look into: What is an artefact? What is an artefact repository? What is the role of artefact repositories in the devops lifecycle? What is an Artefact? Before defining the repository, we must understand the "Artefact" itself. In software development, an artefact is an application or component that has been compiled and built into a single file. Examples include: JAR or WAR files for Java applications. NPM packages for JavaScript. Docker Images for containerized environments. An artefact is designed to be easily movable, shareable, and version-controlled, representing a specific "snapshot" of your code at a point in time. What is an Artefact Repository? An Artefact Reposit...