Posts

(AI/ML): Introduction to Data Engineering

Image
Data has always been dubbed as the "new oil". A very fitting anology. Just as with oil, data is useless  and even dangerous in its raw form.  Just like with oil, which needs to  be refined, transported safely, and delivered to the right place at the right time in order to useful, data must go through the similar process. This is where the Data Engineer comes in. While Data Scientists are like chefs who create a masterpiece meal (the insights and AI models), Data Engineers are the architects and contractors who build the industrial kitchen. They ensure the water lines are pressurised, the electricity is stable, and the ingredients arrive fresh and sorted every morning. In this article, we will have a quick look into:  What is data engineering? what are data pipelines? what is ETL and ELT? why is data quality important? etc. 1. The Core Mission: Building the Pipes The primary responsibility of a data engineer is to build the infrastructure and reliability required f...

(TWN) DevOps Fundamentals: Containerization with Docker

Image
Docker is a open-source platform that has revolutionized how we build, ship, and run applications by introducing a standardised way to package software.  In this article, we will look into:  what is a container? How are docker containers and images different? How are containers and virtual machines different? And deep dive into Docker. 1. What is a Container? At its core, a container is a way to package an application along with all its necessary dependencies, libraries, and configurations. It serves as a portable artifact that can be easily shared and moved across different environments—from a developer's laptop to a testing server and finally to production—without any changes in behavior. Where do containers live? Containers are stored in repositories . DockerHub : The primary public repository for sharing Docker images. Private Repositories : Most companies maintain their own private registries to secure proprietary code. How containers Improve the Development Process?...

(Commentry): Distractions

In the past few days, I had not been at my best. Not been at my 1% improvement everyday. I did not do the things that I said I will do everyday nor come to blog. I did get started on things and tried to study. That was good effort. But, I was distracted.  I have been distracted. It is not being a working adult. It is not easy to adult. It is different from when we were at school or university. We would just think, why not relax. You are getting paid with money, just relax. No harm, no foul.  In university, your timetable was broken down for you, you had semesters. You had assignments. You had exams. It is not the same in the outside, no one has set timetable for you. No one is actually invested in your learning or development. You have to manage it all.  It is easy to think that it is fine to relax. You are still getting paid. But, know this, life and its duties will catch up with you.  Time wasted is time wasted. It is the most important resource that we should safe...

(TWN) DevOps fundamentals: Artefacts & Artefact Repositories

Image
Artefact Repositories  and  Artefact Repository Managers  are critical in the Continuous Integration and Continuous Deployment (CI/CD) of devops. They ensure that we can  store and manage the outputs of our build processes with speed and reliability.  In this article, we will look into: What is an artefact? What is an artefact repository? What is the role of artefact repositories in the devops lifecycle? What is an Artefact? Before defining the repository, we must understand the "Artefact" itself. In software development, an artefact is an application or component that has been compiled and built into a single file. Examples include: JAR or WAR files for Java applications. NPM packages for JavaScript. Docker Images for containerized environments. An artefact is designed to be easily movable, shareable, and version-controlled, representing a specific "snapshot" of your code at a point in time. What is an Artefact Repository? An Artefact Reposit...

(Commentary): The Burnout Cycle: Finding My Way Back to the Game

Being burnt out and losing your way is a universal experience, yet when you’re in the middle of it, it feels incredibly isolating. This post is a personal one. I haven’t blogged in a while because I’ve been navigating a season of significant change. I wanted to take this moment to clear my head, share my truth, and finally get back into the game. Let’s be honest: I lost my momentum. I lost my way, and I lost my consistency. Last week, burnout finally caught up with me, and I spent several days doing absolutely nothing. The habits that usually anchor me—journaling, meditation, studying—all fell by the wayside. When I tried to force myself back into my routine, I realized I didn't even know what I was returning to. I lacked focus. I lacked a "why." I was drifting without a compass. Perhaps it was the weight of recent professional and personal changes. I didn't manage the transition well, and without my usual systems in place, I spiraled into a familiar cycle: a burst of...

(TWN) DevOps Prerequisite: SSH & WinSCP

 As we have learned earlier, being in the field of DevOps require working with servers (Linux servers). Most of these work will also involve working with remote servers. This means that we need a secure and efficient way for DevOps Engineers and Cloud Engineers to talk to the remote servers. That is where SSH (Secure Shell) and WinSCP (Windows Secure Copy Protocol) comes in.  In this article, we will look into:  What is SSH? What is WinSCP? Why is SSH important? How is SSH used in DevOps? What is SSH? Secure Shell (SSH):  - is a cryptographic network protocol that allows secure communication between a client and a server over an unsecured network - is lightweight and efficient, making it ideal for resource-constrained environments What is WinSCP? WinSCP (Windows Secure Copy Protocol): - free, open-source SFTP (SSH File Transfer Protocol) and SCP (Secure Copy Protocol) client for Windows - securely transferring files between a local Windows machine and ...

AI: Intro to RAG

Image
RAG( Retrieval Augmented Generation) is an important component and enabler of efficient Gen AI solutions nowadays. Foundation Models are essentially learned or pre-trained Deep Neural Networks. They can act as the thinking brain but they have some significant disadvantage.   One is "hallucinations" whereby they generate confident but false information,  and the second is the "knowledge cutoffs" (a lack of awareness of events or data after their training period). In this article, we will look into What is RAG? Why do we need it? what advantage do they serve? what is the typical process of a RAG looks like? What is RAG? RAG resolves both these issues by allows the FMs (brain) to ensure that whatever information that it is retrieving and generating is based on grounded organisational document. For example, if we are building a chatbot for a hotel today on our hotel policies, the model itself would not know about the hotel or its policies, building a RAG on top of it wh...