Where R Data Pipelines Pay Their Cost: data.table, dplyr, fread, readr, and vroom
My previous article was somewhat polarizing. Most readers responded positively, but some disagreed with the benchmarking methodology I used. That criticism is precisely why this article exists. To fully understand the pipeline architecture I developed, and why I ultimately chose it over the alternatives, I recommend reading the previous article first: data-table-vs-dplyr-in-a-data-pipeline.html In this new benchmark, I use a more rigorous methodology to produce more precise, reliable, and reproducible results. The goal is not only to compare execution times, but also to make the entire process transparent. Every experiment is designed to be reproducible, and the article can also be read as a practical tutorial for benchmarking this kind of data-manipulation pipeline. The entire raw work behind this article is available here: Article_RSHINY Packages version For those benchmarks, I'll use…
software-computational-architecturedata.table VS dplyr in a data pipeline
Recently, I wanted to create an analytical dashboard for the visits on this blog, entirely server-side. This means reading the NGINX /var/log/statix.log file, reducing bot noise as precisely as possible, and outputting useful visualizations. I know that some libraries, like Streamlit in Python, allow this kind of analysis, but at the time, I was way more comfortable with R. My first version was absolutely awful in terms of performance, not only because I used dplyr, which is a good dataframe engine and noticeably better than base R’s dataframe engine, but also because of a completely awful execution order. In this article, I will not only show you the exact performance difference between the dplyr and data.table implementations, but also show you how much execution order matters in a real data…
software-computational-architectureThe Cartesian Product Disaster Tour: Haskell, C, and 25GB of Allocations
In this article, we’ll explore something that looks innocent: computing the cartesian product of several lists. In Haskell, it is almost boring: Copy ghci> sequence [[1, 2], [3, 4, 5]] [[1,3],[1,4],[1,5],[2,3],[2,4],[2,5]] Clean, elegant, one function call. And that is exactly why it is interesting. High-level languages let us treat this kind of operation as obvious. We call sequence, or itertools.product (Python), and move on with our lives. But under the abstraction, there is still a real data structure being built, memory being allocated, indices being advanced, values being copied, and eventually a CPU doing the dirty work. So I wanted to take this simple operation and push it through several implementations. First, we’ll start with my first Haskell attempt: not idiomatic, a bit painful, and honestly quite funny in retrospect.…
software-computational-architectureWhy Parsers were invented ?
A lot of text-processing tools can get surprisingly far without a real parser. For simple patterns, tools like sed, awk, or small ad-hoc string transformations are often enough. You match a fragment, replace it, split on a delimiter, maybe scan left to right, and the job is done. For flat data, it's often the most direct and elegant solution. But this approach starts to break down when the input has structure. As soon as you have nesting, precedence, escaping rules, ambiguous symbols, or context-dependent meaning, the input should not be considered as just a string. It becomes a small language. At that point, continuing to manipulate characters directly means you are building a parser anyway, just an implicit, fragile, and usually inefficient one. This article is about discovering that boundary.…
software-computational-architectureSAM, the 'other' SED from the 80s
Prerequisites This article will reference SED syntax that is described here: sed This was also vastly ispired by this comment (user: piekvorst) on HackerNews (a lot of credits go to this user and ressources share in the comment): hn SAM - Installation Because SAM is part of Plan 9, we have to install plan9port: Copy $ git clone https://github.com/9fans/plan9port.git $ cd plan9port plan9port$ ./INSTALL then: Copy $ export PLAN9=$HOME/plan9port $ export PATH=$PATH:$PLAN9/bin $ source ~/.bashrc To make it permanent, write the export down to your ~/.bashrc then source ~/.bashrc Where does it come from SAM was written by Rob Pike at Bell Labs in the late 80s. It comes from the same place as UNIX, but from a later phase where people were already trying to rethink some of its…
software-computational-architectureHow Modern Browsers Render the Web: A Deep Technical Journey
From HTML bytes to GPU pixels — the complete architecture of a modern browser rendering engine. Introduction: The Browser as a Game Engine There is a useful analogy for understanding how browsers work at a deep level: think of a browser as a real-time 3D game engine. In a game engine: The scene graph defines what objects exist and where they are in the world. The camera defines what portion of the world is visible. The renderer takes the visible scene and converts it into pixels on screen. The compositor combines multiple render layers (HUD, skybox, geometry, post-processing effects) into the final frame. A browser does the same thing — just with HTML, CSS, and JavaScript instead of meshes and shaders. Game Engine Concept Browser Equivalent Scene graph DOM +…
software-computational-architectureFrom Backtracking to Factorial Coordinates: Understanding Permutations Deeply
Most developers learn permutation generation through recursion and swapping. It works, it's elegant… but it often feels a bit magical. This article goes much deeper. We will: Fully understand how backtracking actually works Build a precise mental model of recursion Discover that permutations form a structured coordinate space Eliminate recursion entirely using factorial number systems (Lehmer code) Derive efficient, controllable, high-performance implementations 1. The Classical Backtracking Algorithm Here is the standard implementation: Copy template <typename T> void permuteHelper(std::vector<T>& vec, int start, std::vector<std::vector<T>>& result) { if (start >= vec.size()) { result.push_back(vec); return; } for (int i = start; i < vec.size(); ++i) { std::swap(vec[start], vec[i]); permuteHelper(vec, start + 1, result); std::swap(vec[start], vec[i]); // backtrack } } 2. The First Misconception A very common intuition is: "A swap creates a permutation" This…
software-computational-architectureBinary Operations: AND, OR, XOR, NOR - and Their Deep Relationships
Binary operations are the foundation of computation. At the lowest level, everything reduces to simple operations on bits: true (1) and false (0). In this article, we explore the core logical operations: AND OR XOR NOR XAND (XNOR) We’ll also look at how they relate to each other — and why XOR can be expressed using only NAND-like constructions. 1. Basic Definitions Let’s define the core operations for two boolean values A and B. A B AND OR XOR NOR XAND (XNOR) 0 0 0 0 0 1 1 0 1 0 1 1 0 0 1 0 0 1 1 0 0 1 1 1 1 0 0 1 Intuition AND ( &&) → both must be true OR ( ||) → at least one is true XOR (…
software-computational-architectureJava Deep Dive: Architecture, Memory, OOP & Modern Features
A comprehensive technical reference — from JVM internals to streams, lambdas, and the philosophy behind Java's design choices. Table of Contents Introduction & Philosophy Everything is a Class Memory Model: Stack, Heap & Metaspace The static Keyword — Java vs C++ Constructors & Method Overloading Encapsulation: Getters & Setters Inheritance, Virtual Dispatch & vtables Overriding toString() — Why It Matters Interfaces, Lambdas & Functional Programming Primitive Types vs Wrapper Classes Comparators as Objects — Everything is a Class Collections, Maps & Sets The Stream API Type Inference in Java Summary Diagram 1. Introduction & Philosophy Java was released by Sun Microsystems in 1995, designed with a clear motto: "Write Once, Run Anywhere". Unlike C or C++, Java programs do not compile to native machine code directly. Instead they compile to…
software-computational-architectureOpenTelemetry in Go, finally explained: traces, metrics, logs, propagation & middleware
I spent the day digging into OpenTelemetry in Go and, honestly, a lot of things that look magical at first finally clicked. The code snippet of this article come from my recreational/tutorial repo: TutOpenTelemetry-Go This article is the version I wish I had when I started. It covers: what context.Context is doing in Go HTTP servers how OpenTelemetry plugs into a Go service what traces, metrics, and logs actually are how otelhttp middleware changes request handling why you can have both a middleware-created HTTP span and your own manual span what propagators do and how they work across services why metrics flush in the background what the stdout trace / metric / log output really means field by field how to read the actual JSON output from the OpenTelemetry Go…
software-computational-architectureA Structured, Visual Algorithm for Determinants up to 5×5
1. Introduction Determinants are fundamental objects in linear algebra, yet direct computation beyond 3×3 can be tedious. This article presents a structured, visual algorithm I designed in two days of focused exploration. It offers a systematic way to compute determinants of matrices up to 5×5 by enumerating 2×2 sub-determinants revealed through a simple column-exclusion scheme. Colored lines in accompanying diagrams denote columns that are “occupied” (i.e., excluded), guiding a stepwise traversal that is easy to follow by hand or implement in code. You can get the algorithm implementation in R here 2. Core Idea: Occupied Columns and 2×2 Submatrices The method is built on a concise principle: at the moment we form a 2×2 submatrix inside an n×n matrix, exactly n − 2 columns are excluded. Graphically, each exclusion is…
software-computational-architectureReimplementing Cross-Validation for KNN in R (from Scratch)
After reimplementing K-Nearest Neighbors (KNN) from scratch, I wanted to take things a step further: build a cross-validation system entirely in base R. This meant re-creating folds, evaluating accuracy across them, and tuning both k (the number of neighbors) and the training proportion. No caret, no external frameworks — just my own code. The Fold Function The first step in cross-validation is creating training/test splits. For this, I wrote v_Rmach_fold, which generates n_fold pairs of train/test datasets. Each sample is stored in a small S4 class sample_Rmach with: train: the training dataframe test: the testing dataframe train_ids: row indices used for training test_ids: row indices used for testing Copy lst_test <- v_Rmach_fold(inpt_datf = iris[1:25,], train_prop = 0.7, n_fold = 4) print(lst_test$sample1@train) # Training set print(lst_test$sample1@test) # Test set This function…
software-computational-architectureReimplementing K-Nearest Neighbors (KNN) from Scratch in R
As part of my exploration of machine learning fundamentals, I decided to reimplement classic algorithms from scratch instead of relying directly on existing libraries. One of the first I tackled was K-Nearest Neighbors (KNN), a simple yet powerful classification method. Why KNN? KNN is often one of the first algorithms taught in machine learning because it’s intuitive: To classify a new point, look at the k closest points in your training data. Assign the most common class among those neighbors. No training phase, no model building — just distance comparisons. This makes it perfect for reimplementing from scratch. The Implementation I called my implementation knn_Rmach. It works with R data frames, allowing you to specify which columns are variables and which is the classification column. It also supports both numeric…
software-computational-architectureOptimizing the Exponential Function in Cherubin
Cherubin is my C++ library for computing very large numbers represented as strings. One of the hardest functions to implement efficiently in this context is the exponential exp(x). A naïve implementation using the Taylor series converges slowly, requires repeated big-number multiplications and divisions, and becomes impractical very quickly. With chexp3, I managed to dramatically reduce the execution time of the exponential function by splitting the input into its integer part and fractional part, and computing them separately. This way, the function avoids huge Taylor expansions and only uses lightweight operations when needed. The Optimization Strategy Here’s the core idea of chexp3: Step 1 — Separate integer and fractional parts: The input x (as a string) is split into partint and partflt. Example: x = "12.34" → partint = "12", partflt…
software-computational-architectureReimplementing the Cauchy Distribution in Fulgurance (C++ Statistics Library)
Introduction Fulgurance is my C++ statistics library where I reimplemented several probability distributions from scratch, including the Cauchy distribution. The goal of Fulgurance is to provide flexible statistical tools in modern C++, without relying heavily on external libraries. It’s also a playground to explore how distributions work internally, both mathematically and computationally. Why Reimplement Cauchy? The Cauchy distribution is a well-known heavy-tailed distribution, often used in physics, Bayesian inference, and robustness testing. Unlike the normal distribution, its mean and variance are undefined — which makes it a good candidate to test statistical implementations. Rebuilding it from scratch in C++ was a great exercise in both numerical computation and random number generation. The Four Building Blocks 1. dcauchy — Density Function The probability density function (PDF) of the Cauchy distribution is:…
software-computational-architectureCheating a Bit: Approximating atan, arccos, and arcsin in the Cherubin Library
This project is part of my Cherubin library — a C++ library for computing very large numbers represented as strings. Because Cherubin works outside the world of hardware floats and doubles, I couldn’t just call std::atan, std::asin, or std::acos. Instead, I had to invent approximations that made sense within the string-based arithmetic of Cherubin. ( GitHub link) The result is a set of small but clever tricks to approximate inverse trigonometric functions. They’re not mathematically perfect, but they get the job done and are fast enough to work with big numbers. Why Approximate? Standard math libraries are precise and optimized for floats, but Cherubin manipulates numbers as std::string or std::deque. Re-implementing full Taylor series everywhere would be very slow, so I turned to shortcuts: polynomial fits, hybrid methods, and even…
software-computational-architectureEmulating Binary-Level Operations and Number Representations
At the heart of every computer lies a simple truth: everything is binary. Whether you are working with integers, floating-point numbers, or characters, all data is ultimately represented as sequences of 0s and 1s. To truly understand how computers operate, developers must go beyond high-level abstractions and study how numbers are represented and manipulated at the binary level. This article covers three pillars of low-level computation: Bitwise operations (AND, OR, XOR, NOT, shifts). Signed integer representations (one’s complement and two’s complement). Floating-point numbers (IEEE 754 standard). 1. Bitwise Operations Bitwise operators act directly on the binary digits of numbers. AND (&) Copy 1101 (13) 1011 (11) ---- 1001 (9) Use case: masking specific bits. OR (|) Copy 1101 (13) 1011 (11) ---- 1111 (15) Use case: setting specific bits. XOR…
software-computational-architectureUnsolved problem !!!
Statement and motivation (inspired from pb.93 of 99 Problems in Haskell) This problem originates from the motivation to find all possible results given a formula with `n` values, like: $$ X_1 + X_2 - X_3 \cdot X_4 $$ So here we are given a set of operators and a set of values. Basically there are n - 1 operators, one between each value. So it is trivial to find all possible results, we just have to use a cartesian product of all operators n - 1 times, then put the operators between each value, and calculate the result. In Haskell, to find all operators combinations, it would look like this: Copy sequence . replicate 3 $ "+-*/" But as you know, in each formula, comes the parenthesis. So it will…
