Kaushik Iska

Why Every PeerDB Cloud Tenant Still Gets a Private Postgres

Kaushik Iska — Fri, 16 May 2025 00:00:00 GMT

I’m Kaushik Iska, co-founder and CTO of PeerDB (YC S23), now part of ClickHouse and powering Clickpipes. PeerDB’s job is simple to describe and hard to build: we sit between an application’s Postgres and the rest of the data universe, streaming rows out in real time so users can analyze fresh data wherever they like. Inside each PeerDB deployment lives a catalog: a Postgres database that records every pipeline definition, credential, usage counter and a small slice of pre-aggregated analytics for quick dashboards. This essay is about that catalog and, more precisely, about the decision we made to give every customer their own logical database inside it.

Choosing the least-regret tenancy model

When the first paying users arrived, we noticed three obvious ways to share Postgres:

Row-level multitenancy looked cheap but came with shared-statistics nightmares and the ever-present risk of one tenant’s bad query affecting everyone else.
Schema-per-tenant felt tidier until you tally up extension drift, pg_namespace bloat and global locks that turn ordinary DDL into a convoy.
Database-per-tenant promised hard isolation.

We picked the last option and parked roughly eighty to a hundred tenant databases on a single RDS instance. (We run several of these instances, but the exact count is private.) We briefly considered a physical Postgres per tenant model (true single-tenant servers), but the economics were brutal for smaller customers, and nothing in our workload justified that burn.

What the catalog actually stores

Because PeerDB merely coordinates data movement, not the data itself, the catalog is light. Every write is either metadata (a new pipeline, a rotated secret) or a counter that jumps once per batch moved. We store those counters already rolled-up so queries stay fast; the hottest table is still well under twenty-five gigabytes. If growth ever forces sharding, we’ll know long in advance, and until then we enjoy the luxury of Postgres doing exactly what it has done well for decades.

Shipping changes without roulette

Our schema lives in a refinery migrations folder. During an upgrade the Kubernetes init-container for each service runs the pending migrations against its tenant databases and holds an advisory lock so one long migration can’t block its neighbors. The process is deterministic, self-healing, and, best of all, boringly repeatable.

Observability

We began with Datadog; the swipe-your-card convenience was unbeatable, right up to the moment the bill wasn’t. Today our observability is powered LogHouse and LightHouse.

What has worked in practice

First, a tiny fleet is easy to watch: a couple of RDS dashboards and a Grafana page tell us everything. Second, deep Postgres expertise pays compound interest. When a query plan misbehaves, someone on the team has already debugged that planner corner case in a past life. Third, our open-source posture matches the model; the public PeerDB repository assumes exactly this catalog layout, so what we dog-food is what we ship.

The trade-offs we live with

Postgres connections are fat. In the early days, before we sized instances correctly, we lived in fear of exhausting the pool. Connection pooling and right-sizing bought back margin, but it was a lesson learned the noisy way. And while automation is good, wrangling hundreds of logical databases remains real work. From permissions audits to extension upgrades, there is no perfect autopilot yet.

Still the best deal in town

Could we outgrow this design? Sure, success comes with new scaling puzzles. But the catalog will tell us long before anything combusts, and the escape hatches (read replicas, horizontal shards, even separate servers for large tenants) are well-trodden Postgres terrain. Until that day, every new PeerDB-Cloud customer gets a freshly minted logical database, a clear blast radius, and the quiet confidence that boring infrastructure lets the interesting work happen elsewhere.

A single RDS instance, many isolated tenant databases: hard walls, clear blast-radius, zero surprise coupling.

See you in the next piece,
Kaushik

Shader Playground

Kaushik Iska — Sun, 26 Apr 2020 00:00:00 GMT

I started reading The Book of Shaders, which I’m thoroughly enjoying. One of the exercises in the book was to replicate the sunset from William Turner - The Fighting Temeraire. Over time, I started to really appreciate the image and the colors in it.

J. M. W. Turner, The Fighting Temeraire (1839), National Gallery, London. Public domain, via Wikimedia Commons.

Before I got started with this, to improve my gradient foo, I decided to go explore creating simpler gradients. Thanks to Adobe Color, I was able to find some of the key colors in the image.

The palette the shaders below use.

Blue to orange diagonal gradient

Linear gradient from top-left to bottom-right blue to orange. In this I tried to capture the sun setting at bottom-right. This is infact a radial gradient.

Pause

Shader source (editable)

Light from the source

If you notice the original image, you will see that Sun isn’t in the bottom right, rather its padded ~25% from both right and bottom.

Pause

Shader source (editable)

Sky getting squeezed

The next thing that caught my eye wat the sky in between the Sun and the water. It looked like it was being squeezed between the two. This reminded my of a hyperbola. After playing around with some hyperbolic parameters. I was able to find something that resembled the blue.

Pause

Shader source (editable)

#ifdef GL_ES
precision mediump float;
#endif

uniform vec2 u_resolution;
uniform float u_time;

const vec3 blue = vec3(0.47, 0.58, 0.71);
const vec3 pale = vec3(0.80, 0.78, 0.70);
const vec3 orange = vec3(0.91, 0.55, 0.22);
const vec3 gold = vec3(0.98, 0.80, 0.42);
const vec3 white = vec3(1.0, 0.97, 0.88);

void main() {
    vec2 st = gl_FragCoord.xy / u_resolution.xy;
    float aspect = u_resolution.x / u_resolution.y;
    vec2 p = vec2(st.x * aspect, st.y);
    vec2 sun = vec2(0.75 * aspect, 0.25);
    float d = distance(p, sun);

// Warm light around the sun, as before.
    vec3 warm = mix(orange, pale, smoothstep(0.0, 0.9, d));
    warm = mix(gold, warm, smoothstep(0.0, 0.25, d));
    warm = mix(white, warm, smoothstep(0.03, 0.06, d));

// The blue gets squeezed between the sun and the water.
    // Level sets of (y - h) * (k - x) are hyperbolas with asymptotes
    // at the horizon and at the right edge, so the blue presses
    // down towards the horizon as it nears the sun.
    float horizon = 0.12;
    float h = (st.y - horizon) * (1.08 - st.x);
    float sky = smoothstep(0.05, 0.22 + 0.01 * sin(u_time * 0.7), h);

vec3 color = mix(warm, blue, sky);
    gl_FragColor = vec4(color, 1.0);
}

Two suns and a river

The next experiment I decided to try was to blend in two radial graidents. One for the sun on the top and the other for the reflection below. Blending these in with the river (modeled as a hyperbola) was the effect that I wanted to achieve.

Pause

Shader source (editable)

#ifdef GL_ES
precision mediump float;
#endif

uniform vec2 u_resolution;
uniform float u_time;

const vec3 blue = vec3(0.47, 0.58, 0.71);
const vec3 pale = vec3(0.80, 0.78, 0.70);
const vec3 orange = vec3(0.91, 0.55, 0.22);
const vec3 red = vec3(0.66, 0.27, 0.13);
const vec3 gold = vec3(0.98, 0.80, 0.42);
const vec3 white = vec3(1.0, 0.97, 0.88);
const vec3 bank = vec3(0.24, 0.21, 0.16);

const float horizon = 0.32;

// Radial glow around c; `stretch` squashes it into an ellipse.
float glow(vec2 p, vec2 c, vec2 stretch, float r) {
    return 1.0 - smoothstep(0.0, r, length((p - c) / stretch));
}

vec3 sky(vec2 st, vec2 p, vec2 sun) {
    float d = distance(p, sun);
    vec3 warm = mix(orange, pale, smoothstep(0.0, 0.9, d));
    warm = mix(red, warm, smoothstep(0.02, 0.5, abs(st.y - horizon - 0.06) * 3.0 + d * 0.5));
    float h = (st.y - horizon) * (1.08 - st.x);
    return mix(warm, blue, smoothstep(0.04, 0.2, h));
}

void main() {
    vec2 st = gl_FragCoord.xy / u_resolution.xy;
    float aspect = u_resolution.x / u_resolution.y;
    vec2 p = vec2(st.x * aspect, st.y);

vec2 sun = vec2(0.75 * aspect, horizon + 0.1);
    // The second sun is the reflection, mirrored about the horizon.
    vec2 reflection = vec2(sun.x, 2.0 * horizon - sun.y);

vec3 color;
    if (st.y > horizon) {
        color = sky(st, p, sun);
        float s = glow(p, sun, vec2(1.0), 0.28);
        color = mix(color, gold, s * s);
        color = mix(color, white, glow(p, sun, vec2(1.0), 0.05));
    } else {
        // The water mirrors the sky, darker, with ripples that break up the reflection.
        float ripple = 0.004 * sin(st.y * 140.0 + u_time * 2.0);
        vec2 m = vec2(st.x + ripple, 2.0 * horizon - st.y);
        color = sky(m, vec2(m.x * aspect, m.y), sun) * 0.75;
        float s = glow(vec2(p.x + ripple * aspect, p.y), reflection, vec2(0.6, 1.4), 0.25);
        color = mix(color, gold, s * s * 0.8);

// The river is bounded by a hyperbolic bank on the bottom left.
        float shore = (horizon - st.y) * (1.25 - st.x);
        color = mix(color, bank, smoothstep(0.1, 0.12, shore));
    }
    gl_FragColor = vec4(color, 1.0);
}

Conclusion

There are definitely extensions to this that would get the resulting render closer to the image, but at this point I felt like I’ll be fine tuning my techniques to get the result closer but not learning as fast as I’d like in the process. I decided to pause this expermient for a bit and continue my exploration.

Towards the end I was satisfied with the effect that the shader resulted in. I enjoyed reading up on coordinate geometry, and also enjoyed staring at Turner’s image for a while. I plan on continuing my experiments with shaders.

Fitness and Freshness

Kaushik Iska — Wed, 25 Dec 2019 00:00:00 GMT

I subscribed to Strava Summit (their subscription based premium offering) as of 6 months ago. Thanks to some friends 2019 has been the best running year of my life. One of the perks of getting the premium is knowing your Fitness & Freshness levels.

Measuring in sports and fitness have fascinated me since I picked up weightlifting and running. So I wanted to get a sense for how individualized fitness is measured using running data.

Banister Impulse Response model

Modeling individual response to stress has been studied by Eric Banister in 1976. The linked paper documents how to measure impulse response in swimmers but was later modified to measure the same in runners and cyclists.

To adapt the model to any stressor (running/swimming/cycling or any exercise), he proposed heart rate based metric called TRIMP. From my understanding, the model works as follows:

Convert each effort into a TRIMP score.
Use the sequence of past TRIMP scores to predict two values:
1. Fitness: The positive influence (due to adaptation and other phenomenon) each effort has on your future performance.
2. Fatigue: The negative effect (due to needed recovery time, muscle soreness etc) each effort has on your future performance.

Fitness outlasts Fatigue

If you think of your efforts over time, each of them will result in a Fitness and Fatigue score as outlined above. You can think of the Fitness and Fatigue curves as the plot of these over time. One of the core postulates of the Banister model is that these curves decay with different half-lifes. Specifically, Fitness has a much higher half-life than Fatigue. What this means is that, if your current Fitness level is 10, it would take ~30 days to go to 0, while the same level of Fatigue would take ~8 days.

This was extremely pleasing to know, given that my workouts during holidays have stalled. Image courtesy Strava.

Performance

Performance is measured as the difference between the Fitness and Fatigue values. It would look something like this.

P(t) = p_0 + e^{-\lambda_{fitness}.t}*k_1 - e^{-\lambda_{fatigue}.t}*k_2

$p_0$ is a constant that represents the baseline performance.
$\lambda_{fitness}$ is the decay of fitness.
$\lambda_{fatigue}$ is the decay of fatigue.
$k_1$ and $k_2$ are coefficients that will be learned.

Given that this is a model with 5 parameters, we would need >= 5 observations to fit. “Form” is used in lieu of “Performance” in Strava.

Conclusion

I hope this understanding of how your body reacts to training impulse and the fact that you will not lose your momentum by missing a couple of workouts in a row, will motivate you towards a stronger, faster and longer 2020.

Happy new year. I’m athlete 11386608 in case you want a buddy to cheer you on!

Scanline Racer

Kaushik Iska — Sat, 07 Dec 2019 00:00:00 GMT

I was recently talking to one of the engineers who worked on Daydream. I realized how little I knew about the challenges in making a VR platform. One thing that struck me as particularly interesting was a rendering technique called “scanline racing”. As someone who is currently working on Graphics, I found it to be fascinating.

Problem

VR has stricter latency requirements than other interactive 3D applications. This is because for humans to feel “immersed” in a virtual scene, the window for an application to render a scene is many times stricter than a typical game. For reference, games generally have latency from mouse movement to screen update of 50 ms or higher (sometimes much higher).

Motion-to-photon latency

The time from when the user moves their head or moves an input device, to the time when that change appears on the display is referred to as motion-to-photon latency. For most users to be comfortable and for VR to feel immersive, the motion-to-photon latency needs to be under 20ms.

Double Buffering is a source of latency

Before we get into how double buffering can introduce extra latency, let’s look at what scanout buffer is and talk briefly about double buffering.

Scanout Buffer

Towards the end, all graphics cards generate a frame buffer, this is a block of memory with 3 bytes per pixel, being the RGB color to show at that pixel on the screen. The frame buffer known to the graphics card is the scanout buffer or the front buffer is the one that will be displayed.

Simple vs Modern Graphics cards

Very simple graphics cards do very little other than supporting a frame buffer and associated output circuitry. They want the operating system to write RGB values into that buffer. Libraries in the operating system then provide more sophisticated APIs for 2-D graphics or 3D graphics, and use software algorithms running on the CPU to compute the color of pixels to be written to the framebuffer.
More sophisticated cards provide various ways for an operating-system to offload the work of mapping graphics operations into pixel-colors from the CPU to the graphics card. They provide SIMD instructions for programmable shaders, but in the end there is a frame buffer that is displayed.

Double Buffering

Traditionally, rendering is double-buffered, which means there are two buffers stored in GPU memory: These are the front buffer and an additional back buffer. At the end of each frame, the two are swapped synchronously. This swap is generally an address change in a register which defines what is the front buffer.

The GPU never renders to the same buffer that’s being scanned out; this prevents artifacts due to potentially seeing parts of an incomplete frame or screen tears.

The effect of increased latency is due to vsync, which causes the front and back buffer swaps to synchronize with the vblank of the monitor. Double buffering itself doesn’t cause this issue and it is useful to minimize flickering because pixels can only change once in the front buffer.

To remove this particular latency single buffering can be used. There is no back buffer, we always render to the front buffer. Once this is in-place we will no longer incur the latency due to the synchronization.

Scanline Racing

To render directly to the front buffer, one has to time things very carefully. The idea us to render each line to the front-buffer before the scanline gets there. This is called “scanline racing” or “racing the beam”. We obviously no longer have a “beam” to race, this word remains from the CRT days.

Based on some reading this technique doesn’t seem to require us to render things line-by-line, we can do this strip-by-strip or any other sequential fragmented method.

This can shave-off a few millis from each scan-line pass, but double buffering existed for a reason. It is much easier to reason about a system which lets you read the whole buffer during the frame interval, that one where you have to precisely calculate the window where the edits are permitted. This is a really cool technique nonetheless.

Here is a slowed-down simulation of both. The blue line is the scanline; each refresh takes 2.4 seconds here instead of 16.7 ms. The white bar is drawn where the input was when that strip was rendered, and the dashed outline is where the input is right now. With double buffering the whole frame is one to two refreshes old by the time it lights up. Racing the beam keeps each strip fresh, at the cost of the bar shearing as the input moves between strips.

Double buffering Scanline racing Pause

Discuss on Hacker News.

References:

Proving Order Axioms for Dedekind Cuts

Kaushik Iska — Sun, 28 Apr 2019 00:00:00 GMT

This week, I started reading Mathematical Analysis I by Zorich after it was recommended by a friend.

I wanted to prove one of the order axioms for Dedekind Cuts, which was left as an exercise to the reader. Who doesn’t love that phrase?

There are a couple of popular ways to construct real numbers. One is using Cauchy Sequences, and the other is using Dedekind Cuts. While the former is also fascinating, let’s stick to the latter for now.

Informally, Dedekind cut is a an open interval towards the left over the number line. I’m being careful not to use the phrase “real number line”, as at this point, the reader doesn’t know what a real number is. We only know what rationals are. This is what a “cut” would look like:

These are of interest, in my case, to prove the consistency of the Axioms for the Real numbers. One way to prove consistency would be to show that there exists some non-empty set, which satisfies all the axioms. In this case we will show that this satisfies the Order Axiom.

Definition

A Dedekind cut is a partition of the rationals Q into two subsets A and B such that

A is not empty.
A is not Q (not all the rationals)
Closed downwards: A contains all number less than every number in A.
A does not contain a maximum.

\begin{aligned} &A \neq \emptyset, \qquad A \neq \mathbb{Q}, \qquad B = \mathbb{Q} \setminus A \\ &\forall a \in A,\ \forall q \in \mathbb{Q}:\ q < a \Rightarrow q \in A \\ &\forall a \in A,\ \exists a' \in A:\ a < a' \end{aligned}

There are a couple of properties that make this set interesting. It is bounded, but does not contain a maximum. It is implied from this definition that B is A complement. Mathematicians do not like to specify more things than absolutely needed.

We will use the following short hand to denote one “cut”. It is important for us convince ourselves that this is indeed a cut. It’s not the subject of this blog post so, i’ll leave it as an exercise to the reader.

x = (A_x, B_x), \quad \text{written simply as } x = A_x \subset \mathbb{Q}

In the above notation it is to be noted that “x” represents one cut, it does not uniquely identify a real. Dedekind Reals would be a set of all such “x”s. Below, we will refer to “x” as a Dedekind Real or real as a shorthand, and set of all Dedekind reals as “R”.

To be proved

\forall x, y, z \in \mathbb{R}:\ (x \le y) \land (y \le z) \Rightarrow (x \le z)

This might seem obviously true on the first look, but remember we don’t even know how to compare reals yet. It’s also to be noted that, we are talking about comparing sets here, not numbers.

Let’s start by defining the order operation for reals:

x \le y \iff A_x \subseteq A_y

Thankfully ZFC allows us to use subset notion. Talk about standing on shoulders of giants. Let’s now define x, y and z.

x = A_x, \qquad y = A_y, \qquad z = A_z, \qquad \text{each a Dedekind cut}

We know that x, y, and z are reals from our definition. Order axiom for rationals states that:

\forall a, b, c \in \mathbb{Q}:\ (a \le b) \land (b \le c) \Rightarrow (a \le c)

We need something that lets us map from Dedekind real space to rational space that lets us use the above axiom to prove order axiom for reals. Let’s prove the following lemma.

Lemma.

x \le y \iff \forall a \in A_x\ \exists b \in A_y:\ a \le b

Proof. ( $\Rightarrow$ ) If $A_x \subseteq A_y$ , every $a \in A_x$ is itself in $A_y$ , so take $b = a$ . ( $\Leftarrow$ ) Take any $a \in A_x$ and pick $b \in A_y$ with $a \le b$ . If $a = b$ then $a \in A_y$ . If $a < b$ , then $a \in A_y$ because $A_y$ is closed downwards. Either way $A_x \subseteq A_y$ , which is $x \le y$ . $\blacksquare$

Now, let’s look at the proposed order axiom for reals.

(x \le y) \land (y \le z) \Rightarrow (x \le z)

Let’s look at the right hand side of the axiom:

Take any $a \in A_x$ . Since $x \le y$ , the lemma gives $b \in A_y$ with $a \le b$ . Since $y \le z$ , it gives $c \in A_z$ with $b \le c$ . By the order axiom for rationals, $a \le c$ . So every element of $A_x$ is bounded by some element of $A_z$ , and by the lemma again, $x \le z$ . $\blacksquare$

It is interesting to note that Cantor and Dedekind formulated the set theory in late 1800s. Zermelo (of ZFC) formulated the axioms even later in 1900s. This is long after calculus was invented. It’s very modern, but mathematicians were freely using real numbers long before then.

It was important to formulate these axioms to counter things like Russels paradox (video).

If you are interested in history of mathematics, this article will be an interesting read.

Navier-Stokes equation for dummies

Kaushik Iska — Fri, 20 Apr 2018 00:00:00 GMT

I was recently on a holiday with a close friend of mine. We started talking about his research in Computational Physics. We were waiting to board the plane back to SFO, and what better time than that to talk about Fluids, given that most popular forms of long distance travel are through air and water.

Very recently my understanding about how aeroplanes worked was debunked. My previous understanding was that it was due to the lift caused by the difference in pressure above and below a curved surface. Also known as Bernoulli’s Principle. This is incorrect. For the correct explanation, refer: Nasa Article and Dr. Babinsky’s Journal article.

Curious about what equations govern water and more generally fluids, I asked him to, in a few minutes, give me a better model to think about fluids. Following is an attempt at summarizing what I understood:

Kaushik, what do you know about solids?

Continuity Equation: Mass cannot be created nor destroyed.
Newton’s second law: F = ma, otherwise known as Newton’s second law
Conservation of energy: Energy cannot be created nor destroyed.

What makes fluids complex is interaction between the layers. We end up with a modified version of the momentum equation. This modified version should at least account for:

Forces between the layers of fluids: Pressure, Viscosity
Energy and its transfer: Thermal, Potential, Kinetic

The Navier-Stokes equations are an expression of Newton’s Second Law for fluids, stating that mass times the acceleration of fluid particles is proportional to the forces acting on them. If we take the Navier-Stokes equations for incompressible flow as an example, which we can write in the form:

\rho \frac{D\mathbf{u}}{Dt} = -\nabla p + \mu \nabla^2 \mathbf{u} + \mathbf{f}

Navier-Stokes Equation (incompressible form)

The L.H.S is the product of fluid density times the acceleration that particles in the flow are experiencing. This term is analogous to the term m a, mass times acceleration, per the momentum equation. R.H.S has forces that are responsible for particle acceleration:

Pressure gradient
Viscous shear stresses
Volume forces. If the volume forces are due to gravity alone, then we have f = ρg, where g stands for the gravitational acceleration vector.

It’s interesting to note that this equation is very difficult to solve*.*

Obama @ MIT. Navier-Stokes equation.

Hopefully you have a slightly better understanding of how fluids are modeled.

Cutting Buildings

Kaushik Iska — Mon, 09 Apr 2018 00:00:00 GMT

When building high quality POI (points of interest) maps, one thing that a consumer would want is high quality polygons for the places. Businesses keep moving, listings keep changing; buildings are relatively static in this otherwise dynamic world.

When you think about streams of data coming, you’d be wise to account for noise. Specially when it comes to POI data. Let’s start by looking at a couple of malls:

"name","wkt"
"Hillsdale Mall","POLYGON((-122.30062655184946 37.53862272401262,-122.30159214709482 37.539371367343186,-122.30232170794687 37.538826900211824,-122.30206421588144 37.53858869459169,-122.30240753863535 37.53834198082578,-122.3017316219636 37.537780491281076,-122.30180672381601 37.53756780474339,-122.30135611270151 37.53714242984815,-122.3010664341279 37.53715093736983,-122.29973605845652 37.53600241316314,-122.29934982035837 37.53598539785674,-122.29795507167063 37.5370658621093,-122.30010083888254 37.538928988101794,-122.30062655184946 37.53862272401262))"
"99 Ranch Mall - Milpitas","POLYGON((-121.91881917933222 37.42364933052066,-121.91771410921808 37.42397310706446,-121.91720985392328 37.42377713721835,-121.91687726000544 37.423206265612926,-121.91661976794 37.42328295011149,-121.91623352984186 37.42266947192397,-121.91650175074335 37.4225587044933,-121.9157626503088 37.421409801367815,-121.91578410798093 37.420847432486816,-121.91685571711554 37.42039327456982,-121.91712393801703 37.42075114833499,-121.9162549022962 37.42110049964655,-121.91767110865607 37.423554433656996,-121.91867961924567 37.42326473730071,-121.91881917933222 37.42364933052066))"

If you are wondering what wkt means. Wikipedia says:

Well-known text (WKT) is a text markup language for representing vector geometry objects on a map, spatial reference systems of spatial objects and transformations between spatial reference systems.

It’s essentially a way of representing 2-D geometries. In our example, we only need to bother ourselves with POLYGON. It’s a list of (x, y) co-ordinates which correspond to (longitude, latitude) pairs. A convention that a lot geospatial libraries use is longitude-first. To visualize the wkt you can visualize it at this website.

As I have said before, these are the relatively static components, buildings rarely change. The things that do change often on the other hand are POI data, here’s a made up example:

POI Name	Latitude	Longitude	Parent Building
Levi’s	37.1121	-127.112331	Hillsdale Mall
Cream Stone	37.11301	-127.112661	Hillsdale Mall
Uniqlo	37.11101	-127.112371	Hillsdale Mall
99 Ranch	37.44623	-126.1111233	99 Ranch - Milpitas
Delhi Masala	37.44618	-126.111027	99 Ranch - Milpitas

Let’s take a look at how this would look like, with more (near real-life) number of places inside a mall:

The task at hand is to divide the building to accommodate the POI.

Voronoi diagram

One strategy that seems intuitive to start with, is that each point should own the area closest to it. This is where Voronoi comes in. It does exactly this.

In mathematics, a Voronoi diagram is a partitioning of a plane into regions based on distance to points in a specific subset of the plane. That set of points (called seeds, sites, or generators) is specified beforehand, and for each seed there is a corresponding region consisting of all points closer to that seed than to any other.

There are a lot of applications for Voronoi. It’s awesome because:

It’s relatively fast O(nlog n) if we use Fortune’s Algorithm.
Not a lot of edge cases to handle.
Fair splits for points.

Let’s take a look at how the result looks like, if we apply Voronoi on our synthesized POI in buildings.

Does it mirror the real world?

While Voronoi cuts up the building in a reasonable way, it has a major drawback. It does not reflect how the building plans are laid out in the real world. This is because, it does not take the shape of the building into consideration. One of my colleague described the cut buildings as being very Picassoesque.

So, we need to think of this problem with some real world consideration. Data that we get from suppliers is best-effort. It is still noisy. So sometimes it make sense for us to start fresh.

Blank Slate

Let’s start with a blank slate, in our case, blank buildings. Ignoring the points for a seconds, let’s think about the following question: If you were to accommodate N stores inside the buildings. Assuming all the stores are the same (and no elevation), how would you divide the building?

I think a good place to start is splitting the building into block shapes, and minimizing the unused area. This is definitely easier said than done for a general building. We’ll start and take a look at how involved the process is.

Algorithm Outline

We first need to find how the building is oriented, for this we use PCAPCA
We then try and cut the building into NN pieces, where we try to increase the mean area of each lot, under the constraint tha cuts can be made only parallel to the PCA-axes.

PCA

PCA finds a set of dimensions such that they are orthogonal and ranked according to the variance of data along them. This means that, more important principle axis occurs first (more important = more variance/more spread out data).

So, if we find the PCA for the building outline, we get a set of axes that we can then make blocks on. Let’s look at the axes that we find for our buildings.

After the transformation, you can use dynamic programming to divide the polygon into contiguous blocks to allocate to the POI. You can see visualize it as the grids between the points.

Piece-wise linear regression

One other interesting experiment we’ve tried is to use spline or piece-wise linear regression on the points to see if placing the centroids of the POI on the line segments would lead to interesting outcomes. The extension to this experiment that I’d like to try is to buffer the boundaries of polygons inwards to see if that leads to better fits. It makes sense as we’d like to favor segments be inside the polygons rather than outside as you will see below.

Note: The blue line is the regular linear regression line.

You seldom operate with perfect data, it’s critical to salvage the most of what you have. Hope you had fun reading this.I will be uploading the code to my github soon. Let me know if you think of any other techniques to tackle this problem.

A lot of ideas here are thanks to my colleagues at SafeGraph.

Finding duplicate places in your data

Kaushik Iska — Fri, 16 Mar 2018 00:00:00 GMT

Dealing with duplicate places is a challenging task. If you look at standard ways of dealing with duplicates, they revolve around string similairty algorithms, like Levenshtein, Jaccard, Jaro-Winkler and so-on.

There’s a general task of finding duplicate records in databases. That involves dealing with things at a different scale than traditional string comparision algorithms handle. Dealing with all potential matches pair-wise is going to be extremely slow. There is a class of hashing algorithms that are good at dealing with such tasks. They are known as LSH or Locality-sensitive hashing. LSH is a powerful technique with many cool applications - this article by Uber engineering is one such.

What all these methods have in common is that they are relatively stateless. This is not needed for string comparision or data duplication detection. They do not have any idea what part of a word provides more context, and what is not really useful. This causes some issues when we are dealing with duplicate places as you shall see.

Let’s look at a few examples of why this task is challenging:

#	Place 1	Place 2	Reason
0	Emelina’s	Emelina’s Peruvian Restaurant	Large edit distance
1	Hillsdale Mall Levi’s Outlet	Hillsdale Mall Gap Outlet	High string similarity but these differ in key terms.
2	San Francisco Airport	San Francisco Airport Terminal - 1	Nested places are hard.
3	Central Park	Central Park Cafe	In New York City, near Central Park, These may well be different. In most other places, they are almost certainly the same.

To solve this problem, the engineers at Facebook wrote an excellent paper - Deduplicating a Places Database. I’ve prototyped the algorithm outlined in a section in the paper and will walk you through some of the results I’ve seen.

To be more precise, this is the problem we are trying to solve:

Given two place names $n1$ and $n2$ , and an approximate geographical location containing both of them, determine if the two names can refer to the same entity.

The solution sketch to this problem has three main components.

The first is a novel generative language model for places that identifies the core of a place name.

For example in “Emelina’s” vs “Emelina’s Peruvian Restaurant”, it identifies “Emelina’s” as the core and “Peruvian”, “Restaurant” as the background words. More on this later.

The second component is a language model that incorporates the spatial context, e.g., the presence of nearby land- marks, parks, malls, airports, major streets and the current city.

The spatial model helps us with distinguishing “Hillsdale Mall Levi’s Outlet” from “Hillsdale Mall Gap Outlet” and matching/non-matching Central Park and Central Park Cafe.

The third component of our solution is a place name similarity measure based on our language models.

I’m only going to talk about the first part. I’ll try and write follow up posts for the others.

Approach for the Simple Name Model

Let’s build some intuition by starting with an example. Consider the set of $names$ to be:

Philz Coffee
Starbucks Coffee
Philz

Our goal is to come up with core words for each of these words. The result should be:

Philz
Starbucks
Philz

This is because Coffee is a “Background” word.

Let’s define:

$vocabulary$ - set of all tokens/words in our $names$ .
$C(w)$ - Probability that token $w \in vocabulary$ is a core word.
$B(w)$ - Probability that token $w \in vocabulary$ is a background word.
$z(w, n)$ - Probability that token $w \in n$ | $n \in names$ , is a core word.

Algorithm

We start with $vocabulary = \{Philz, Coffee, Starbucks\}$ . $C, B$ are uniform distributions. This looks like:

C[Philz] = C[Coffee] = C[Starbucks] = 1/3
B[Philz] = B[Coffee] = B[Starbucks] = 1/3

Thus, we have

$z(Philz, n1) = z(Coffee, n1) = 0.5$
$z(Starbucks, n2) = z(Coffee, n2) = 0.5$ , and
$z(Philz, n3) = 1$

Based on this, we will update our $C$ and $B$ values.

$C = \{Philz ← 1/2,Starbucks ← 1/6,Coffee ← 1/3\}$ and
$B = \{Philz ← 1/4,Starbucks ← 1/4,Coffee ← 1/4\}$

If you notice, the core probability of Philz and the background probability of Coffee increases. The algorithm converges to the following solution:

$C = \{Philz ← 2/3, Starbucks ← 1/3,Coffee ← 0\}$
$B = \{Philz ← 0, Starbucks ← 0, Coffee ← 1\}$

The algorithm drives down the core probability of Coffee to 0 and background probabilities of Starbucks and Philz to 0.

eps = 1e-16

# this is to get map from token_number to word
def get_inverse_vocab_map(count_vect):
    invert = dict((v, k) for k, v in count_vect.vocabulary_.items())
    return collections.OrderedDict(sorted(invert.items()))


def compute_core_and_background(names, verbose=False):
    count_vect = CountVectorizer(stop_words='english', analyzer='word')
    word_vec = count_vect.fit_transform(names).toarray()

    invert = get_inverse_vocab_map(count_vect)
    w_T = word_vec.T

    n_words = len(count_vect.vocabulary_)
    n_names = len(names)

    C = np.full(n_words, 1.0 / n_words)
    B = np.full(n_words, 1.0 / n_words)

    Z = np.zeros((n_words, n_names))
    _round = 1

    while _round <= max(n_names, 100):
        _round += 1

        min_bg = np.min(B)
        # until convergence do us part
        if min_bg < eps:
            break

        amplified_coreness = C / B
        K = (w_T.T * amplified_coreness).T
        Z = K / np.sum(K, axis=0)
        C = np.sum(Z, axis=1) / np.sum(Z)
        Z_prime = w_T - Z
        B = np.sum(Z_prime, axis=1) / np.sum(Z_prime)

    # avoid division by zero
    B += eps

    if verbose:
        res = []
        for k, v in invert.items():
            k = (v, "core:", C[k], ", bg:", B[k])
            res.append(k)
        res.sort(key=lambda x: -(x[2] / x[-1]))
        for i in res:
            print(i)
    return count_vect, C, B

Let’s run this on our example:

_ = compute_core_and_background(
    ['Philz', 'Starbucks Coffee', 'Philz Coffee'], True)

('philz', 'core:', 0.6666666666666666, ', bg:', 1e-16)
('starbucks', 'core:', 0.3333333333333333, ', bg:', 1e-16)
('coffee', 'core:', 1.0087745832662039e-17, ', bg:', 1.0)

Analyzing the points of interest in Palo Alto

For this exploration. I’m using the sample dataset of places in Palo Alto, filtered to Zip Code: 94301.

Note: I’ve added some noise and duplicates to this dataset to make this effort more interesting.

palo_alto = pd.read_csv('https://gist.githubusercontent.com/iskakaushik/8a1c7580dd8f5ba2841bb009c28dbe76/raw/7d8aab8731e717bdbc388b1faa04af072ecb9798/palo-alto-places.csv')
names = palo_alto.name.tolist()

count_vect, C, B = compute_core_and_background(palo_alto.name.tolist())
palo_alto.head(3)

#	name	city	zip	latitude	longitude
0	wilby optical	palo alto	94301	37.441920	-122.141880
1	deborahs palm	palo alto	94301	37.450087	-122.160125
2	scandia home	palo alto	94301	37.439012	-122.159034

invert = get_inverse_vocab_map(count_vect)
core_word_identified = []

for name in names:
    one_hot_name = count_vect.transform([name]).toarray()[0]
    coreness_vector = one_hot_name * (C / B)
    hottest_i = np.argmax(coreness_vector)
    hotness_score = np.max(coreness_vector)
    core = invert[hottest_i]
    core_word_identified.append((core, name, hotness_score))

core_words_df = pd.DataFrame(core_word_identified, columns = ['core_word', 'name', 'score'])
core_words_df.head(3)

#	core_word	name	score
0	optical	wilby optical	1.696486
1	deborahs	deborahs palm	1.696486
2	scandia	scandia home	1.696486

Score Distributions

Left is unscaled scores. Right ranks scores, to make the plot tighter.

Finding duplicates:

We identify the words that share the same core_word. We can tune the $score$ threshold to pick one that we are confident will result in strong matches. We can then deal with the duplicates by picking one or merging. We can use the score distribution to arrive at reasonable thresholds.

As you can see, we’ve identificed $22$ high condent matches.

THRESHOLD = 5.0

duplicates = core_words_df.groupby('core_word').filter(lambda x: len(x) >= 2).copy()
duplicates['rank'] = duplicates['score'].rank(method='min')
ranked_duplicates = duplicates.sort_values(by=['rank', 'core_word'], ascending=False)

high_score_ranked_dups = ranked_duplicates[ranked_duplicates.score > THRESHOLD]

print("%s high confident matches." % len(high_score_ranked_dups))

high_score_ranked_dups

22 high confident matches.

#	core_word	name	score	rank
53	coolsculpting	coolsculpting	4.259851e+13	422.0
314	coolsculpting	coolsculpting	4.259851e+13	422.0
649	coolsculpting	coolsculpting	4.259851e+13	422.0
810	coolsculpting	coolsculpting	4.259851e+13	422.0
75	walgreens	walgreens	2.129925e+13	414.0
415	walgreens	walgreens	2.129925e+13	414.0
613	kerastase	kerastase	2.129925e+13	414.0
802	kerastase	kerastase	2.129925e+13	414.0
99	cvs	cvs	2.129925e+13	414.0
779	cvs	cvs	2.129925e+13	414.0
37	chase	chase	2.129925e+13	414.0
199	chase	chase	2.129925e+13	414.0
41	mobile	spring mobile	5.089457e+00	404.0
724	mobile	t mobile	5.089457e+00	404.0
475	lemonade	lemonade la	5.089457e+00	404.0
704	lemonade	lemonade	5.089457e+00	404.0
171	footwear	tone footwear	5.089457e+00	404.0
722	footwear	footwear etc	5.089457e+00	404.0
151	comcast	comcast	5.089457e+00	404.0
714	comcast	comcast ventures	5.089457e+00	404.0
454	block	h and r block	5.089457e+00	404.0
776	block	block advisors	5.089457e+00	404.0

Let’s take a look at some of the duplicates we have identified.

Not all duplicates are good

Here are some examples of matches on core words that had a low confidence score. Compare $0.6$ to $5$ . Art also lies in data exploration and picking the right thresholds.

#	core_word	name	score	rank
625	alto	palo alto law horn and mantis kung fu	0.561628	5.0
365	alain	anna slutsky alain pinel realtors	0.506742	3.0
788	alain	alain pinel realtors	0.506742	3.0
573	service	a jag tax preparation and bookkeeping service	0.481377	1.0
584	service	gonzales juanita a ea tax preparation and bookkeeping service	0.481377	1.0

Hope you had fun reading the post.

Thanks,

Kaushik Iska

Structures in jemalloc

Kaushik Iska — Thu, 05 Oct 2017 00:00:00 GMT

Times have changed. In modern systems, spatial locality of memory is more important than minimizing the total number of pages. Data that is allocated together is often used together. It is also optimized to reduce lock contention when multiple threads try to allocate memory.

jemalloc like every other malloc implementation, needs to support two primary API calls:

Allocating memory
- malloc: returns memory chunk
- calloc: returns zeroed memory chunk
free: Freeing allocated memory

Does OS not do this for us? Short answer, No. OS gives us two ways of requesting memory:

mmap: maps device space memory (generally file system) to user space memory.
brk: is used to change the program break address (which is the first location after the end of the uninitialized data segment). Increasing the program break allocates more memory, decreasing deallocates memory.

Structures

jemalloc uses the above two ways to get a memory that it will then manage for the user. To do this it uses various structures. The key structures are:

Arena

This is the highest unit of memory management structure that jemalloc uses. If we imagine a pool of arenas, each thread is given an arena based on the hash of its thread-id or a similar mechanism. This is to prevent lock contention, in an ideal case, if each thread has it’s own arena, there would be no contention.

class Arena {
  List<Thread> threads; // Threads that this arena is associated with
  List<Chunk> chunks; // Chunks that this arena manages
  Lock lock; // guard accesses to various things in this arena.
  RBTree<Run> runsAvailClean; // let's talk about runs in a little bit.
  // clean vs dirty = free zeroed data (calloc) vs free dirty data (malloc).
  RBTree<Run> runsAvailDirty;
  List<Bin> bins; // we will look at Bins soon.
}

Chunk

Chunks are virtual memory areas that available memory is conceptually divided into. They are typically of the same size: 2MB. Each chunk is associated with an Arena. Chunks are the highest abstraction used in jemalloc’s design, the rest of the structures described below are actually placed within a chunk somewhere.

class Chunk {
  Arena arena; // Arena that this chunk is associated with
  boolean dirtied; // this is set if the chunk has ever been dirtied
  List<Run> runs;
  // metadata about free pages for a region size
  RBTreeMultimap<RegionSize, Page> freePages;
}

Regions

A region is jemalloc term for the size of the allocation a user requests. Regions are divided into three classes according to their size, namely:

small/medium: these regions are smaller than the page size (typically 4KB)
large: these regions are between small/medium and huge.
huge: size of these is greater than the chunk size. These are dealt with separately and not managed by arenas, they have a global allocator tree.

enum RegionSize {
  SM_2(2),
  SM_4(4),
  SM_8(8),
  SM_20(20),
  SM_40(40),
  .
  .
  .
  LARGE_2MB(2 * 1024 * 1024);

  int allocationSize;
}

Run

In essence, a chunk is broken into several runs. Each run is actually a set of one or more contiguous pages (but a run cannot be smaller than one page). Therefore, they are aligned to multiples of the page size.

class Run {
  Bin bin; // bins are further down in our discussion
  RegionSize regionSize; // the region size for this run.
  List<Page> pages; // pages that are a part of this run.
  int numFreeRegions;
  int totalNumRegions = pages.size() / regionSize;
  boolean freeRegions[totalNumRegions]; // initially all set to true
  // more metadata to quickly identify the first available region
}

The main responsibility of a run is to keep track of the state (i.e. free or used) of regions. Each run holds regions of a specific size and their state is tracked with a bitmask. Note: I simplified the bitmask to an array of booleans for clarity.

Bin

A bin is what keeps track of free regions of a specific size. All the runs in a Bin have the same region size, equal to the binRegionSize. Every Run is part of some Bin. A Bin can manage multiple Runs.

class Bin {
  RegionSize binRegionSize;
  List<Run> runs;
  Run currentRun; // current run of this bin
}

Picture Time

Example Allocation

Let’s start with a single call to malloc(2). Assuming that the smallest RegionSize we support is 2 (this is a pretty common smallest value) which corresponds to SM_2.

The Arena would start with no chunks and one Bin for every RegionSize. All the bins would start empty. The bin at index 0 in Arena.bins, would then end up with one Run, with one of its first available Page.region allocated to the caller of malloc(2). The Bin’s currentRun would be set to this Run.

If there was a follow up malloc(4) . We would to the same but for the bin at index 1 in Arena.bins. This would also have exactly one Run.

This hopefully gives you some insight into how jemalloc works. As I dug through this, i’ve noticed that theres a lot more subtleties to memory management thatn I first imagined.

References:

Datastructures for Analytics

Kaushik Iska — Thu, 14 Sep 2017 00:00:00 GMT

If you know me, you probably heard me go on about data structures and anime. I’m a Software Engineer who works primarily on systems problems. It’s not rare to see some really cool data structures in action, but not as often as I’d like.

I am writing about a couple that caught my attention.

Cardinality using HyperLogLog (HLL)

Cardinality refers to the uniqueness of data values contained in a particular column (attribute) of a database table. The lower the cardinality, the more duplicated elements in a column.

For people familiar with database queries, it’s essentially a SELECT COUNT(DISTINCT) query.

Some queries that can be answered by cardinality:

Number of unique DNS requests from a given IP
Number of unique visitors throughout the history for google.com

The naive approach of running the SQL on a non-distributed database will of course not scale. Let’s say we store the data in files across multiple boxes, e.g. HDFS, we could run a map-reduce job to count the number of unique entries. If you have ever used hadoop/spark, you know that it is not going to be realtime.

Enter HyperLogLog.

Let’s make the problem simpler. Let’s say we have a stream of integers coming in. At any given instant we wish to know how many unique numbers we have seen. It’s easy to see that in random data, a sequence of k zero bits will occur once in every 2ᵏ elements, on average. All we need to do is look for these sequences of long trailing zeros and record the length of the longest sequence to estimate the total number of unique elements.

What if the first number I see has 32 trailing zeros?

To account for this we could hash each number of the input stream multiple times, count the trailing zeros and take the mean.

HLL Mean Trailing Zeros

This approach has one nice property. For a 2^k bit number you only need log2(k) bits to record the maximum number of traling zeros. Hence the name HyperLogLog. This idea forms the basis for HLLs.

Refer Further readings for more details.

Top K using Count-Min (CM) Sketch

“Sketching” data structures store a summary of a data set in situations where the whole data would be prohibitively costly to store

An example “sketching” data structure could be a Bloom Filter or even HLL (you’ll notice how little space HLLs were taking up).

Let’s talk briefly about Bloom Filters. Bloom Filter is essentially a set, that can say if an element is present with a chance for false positives.

Adding an element to a Bloom Filter:

Have a bitset initially set to all 0 .
Compute K hashes for the element: v1 v2 v3 v4 v5 ... vk.
Mark all the indexes at vi to 1 .

To check for the membership of an element. You’d compute the hashes again and check if all of those bits are set. With the right hash functions and a high enough capacity for the bitset, one can greatly minimize the false positive percentage. They are used extensively in Cassandra.

CM works very much like a Bloom Filter. We have k hash maps, each with a different hash function (note that the Bloom filter can be implemented in this way as well). They map from the hash value of the element to the frequency. Initially we have all frequencies set to 0. When we encounter an element, we increase it’s value in all the k hash maps. If a decrease operation is allowed (which makes things more difficult), we similarly subtract a value from all k maps.

To obtain the count of an element, we take the minimum of the k fields that correspond to that element (as given by the hashes). This is why it’s called a Count-Min sketch**.**

If you want to play around with these data structures, I recommend looking at addthis/stream-lib: Stream summarizer and cardinality estimator., it was also interesting digging into the source code.

Coming up next:

Percentile Summarization using T-Digest
Any other recommendations?

Cryptography Engineering: Introduction to Cryptography

Kaushik Iska — Sun, 22 Nov 2015 00:00:00 GMT

This chapter aims to give us the background necessary to understand the rest of the book.

Encryption. “Encryption is the original goal of cryptography”. Interesting. I am not sure what other uses for cryptography are. (Maybe authentication?). They start by introducing Alice, Bob and Eve to us. Eve is the eavesdropper, Alice and bob try to communicate without Eve finding out.

“An attacker shouldn’t be able to learn any information about m, except possibly it’s length and the time it was sent”

We skip key exchange for now and assume that Alice and Bob have ways to exchange keys, i am guessing this is an infrequent event. Alice sends Bob the cipher text after encrypting message m with the encryption key K_e.

Kerckhoffs’ Principle. Kerckhoffs, a french dude, wrote two journal articles on La Cryptographie Militaire, in which he stated six design principles for military ciphers. The summary of them being:

“the security of an encryption scheme should depend only on the secrecy of the key, and not on the secrecy of the algorithm.”

Authentication. This is different from encryption. How does Bob know that the message he got was from Alice, this is totally possible if Eve has the ability to manipulate the channel. We now introduce a new key K_a, which is different from the encryption key. We introduce a new term, message authentication code or MAC.

Alice computes a = hash(m, K_a), and sends both m and a
Now Bob authenticates by computing a himself.

“Pure authentication is only a partial solution”. Eve can record some messages and can replay them on the channel. Eve can also delete messages. Naughty Eve. Well, message numbering can solve this problem. This still leaves the possibility of deleting the messages and delaying them but thats as far as she can go.

Public-Key Encryption. How do we exchange keys. Alice and Bob meet in person, they exchange their key. What is 20 people meet and decide to exchange keys, there’d be 200 pairs of keys.

Kaushik Iska

Why Every PeerDB Cloud Tenant Still Gets a Private Postgres

Choosing the least-regret tenancy model

What the catalog actually stores

Shipping changes without roulette

Observability

What has worked in practice

The trade-offs we live with

Still the best deal in town

Shader Playground

Blue to orange diagonal gradient

Light from the source

Sky getting squeezed

Two suns and a river

Conclusion

Fitness and Freshness

Banister Impulse Response model

Fitness outlasts Fatigue

Performance

Conclusion

Scanline Racer

Problem

Motion-to-photon latency

Double Buffering is a source of latency

Scanout Buffer

Simple vs Modern Graphics cards

Double Buffering

Scanline Racing

References:

Proving Order Axioms for Dedekind Cuts

Definition

Navier-Stokes equation for dummies

Cutting Buildings

Voronoi diagram

Does it mirror the real world?

Blank Slate

Algorithm Outline

PCA

Piece-wise linear regression

Finding duplicate places in your data

Let’s look at a few examples of why this task is challenging:

Approach for the Simple Name Model

Let’s define:

Algorithm

Analyzing the points of interest in Palo Alto

Score Distributions

Finding duplicates:

Not all duplicates are good

Structures in jemalloc

Structures

Arena

Chunk

Regions

Run

Bin

Example Allocation

References:

Datastructures for Analytics

Cardinality using HyperLogLog (HLL)

Top K using Count-Min (CM) Sketch

Coming up next:

Further readings:

Cryptography Engineering: Introduction to Cryptography