I was the designated cameraman ^™ for the roommates' Seattle trip (largely through self-imposition, for the record). We went camping for three days – the first was spent packing/road-tripping to the campsite, the second was spent hiking, and the third was spent kayaking. I mostly took pictures on the second day because I was too scared to take a new camera into the middle of a lake, and the pool of water in my kayak asserted that to be a good decision. I did get some fun shots on our way into the national park though.

Amateur tip: sitting shotgun and setting shutter speed to 1/2000 or higher can get you some really nice pictures on scenic road trips, clean windshield permitting.

I was pretty shocked when we hit our first viewpoint on the drive in. Apparently water can turn this tealish-green color because of glacial sediment – the more you know!

This was also the closest I'd been to a snow-capped mountain in a long time, so I spent a solid amount of time staring up at those.

The main event was a loop of the Maple Pass trail. The route is absolutely beautiful, albeit a bit steep for sedentary grad students (I wonder who that could be?). I started feeling a solid amount of struggle about a third in, but the first lookout was a much-needed shot in the arm.

This picture is somewhat emblematic of some of the struggles I had taking pictures throughout the hike. I wanted to get as much of the landscape as possible, which is inherently a very deep thing to shoot, so I resorted to vertical shots a lot. At the same time, properly exposing the snow, greenery, and rock faces all at once was a tall task, and not one I'd read the manual enough at the time to tackle. So there are quite a few ethereally blown-out snowcaps and clouds.

I did, however, get this shot of my friends taking a break which I thought was pretty cool. You can once again see that exposure was a challenge.

It's not super clear from the pictures because I wasn't taking a lot of them initially and didn't want to hold up my friends, but the first part of our hike was through a really twisty forest section. At some point we broke out of the forest and entered a section I lovingly referred to as "Lord of the Rings" because of the way the greenery and lighting looked. At this point I also received reassurance that it was okay if I hung back a little and took my time taking pictures, so I got some decent shots of the terrain while we hiked.

A little farther into lord-of-the-rings, and we started to see some pretty huge rock faces.

We didn't realize it at the time, but we hit the best lookout/vantage point a little bit after that. It's a little hard to recollect exactly how I felt seeing this with my own two eyes, but quite a few "holy shits" and "oh my gods" were muttered while sitting on a rock.

This was also about the point that we realized that we still had about half the hike to go (a little more upwards then the entire downclimb), so we all picked up the pace a little at that point. As a result, the number of pictures taken declined rather rapidly, but there are a couple shots that I'm glad I took during the hurry.

As an aside, we'd mostly been hiking through in tshirts and pants by this point. So staring at what seemed to be a scene straight out of an Indian movie's Europe shooting while wearing my standard work attire was pretty crazy.

This shot has a snakey leading line going through it that I liked. Plus my friend using his hiking stick as a golf club was kinda funny.

And arguably my favorite shot from this trip:

In hindsight I might have cropped off a little more of the sky, but it gives the people some breathing room, which I kind of like. Coincidentally this was also the last decent shot I took of our hike, since the footing was quite unstable during the downclimb and we were really trying to beat the light at that point. Case in point, here's a weirdly exposed shot of the sunset light before we'd even made it back into the forest.

If it wasn't already apparent, we made it back before dark, and no major issues were had getting back to our campsite. The kayaking trip the next day was just as enjoyable, if a little less picturesque, though there are fewer visual records of that excursion.

If I were to summarize this post so far:

TL;DR: X-Pro 3 good, did not know how to use it fully during the trip so pictures were less than ideal, North Cascades are absolutely beautiful.

I'd actually bought the camera right before we started driving, so I didn't have a ton of time to figure out the features of it that were different than my X-T20 and generally didn't have a great feel for how to best take advantage of it. The weekend after I got home, I finally sat down and read through the entire manual, configuring all the settings in a way that felt intuitive to me. This helped feel more comfortable when I went out on photowalks, and I've been taking pictures around campus that I'm much more satisfied with from a quality perspective.

This post was initially intended to be a super-cut of all the pictures I'd enjoyed taking this month, including the improved ones from when I got back to campus, but the hike section ended up being way longer than I'd initially anticipated! Plus it's kinda fun recounting the experiences and thoughts that came along with the shots.

I'm hoping to make a second post sometime in the next few weeks about the pictures I've taken since coming back to campus, since I had a bit more creative freedom when taking those to play with framing and whatnot.

In the meantime, I hope this post was some mixture of informative and entertaining. It's a beautiful world out there, and we ought to take more pictures of it!

The classical advice goes that you prefer to keep processes on the same core for as long as possible (without inducing unfairness) so that you can keep their data warm in the caches, and Linux's EEVDF scheduler does a pretty good job of that. What isn't taken into account is making sure processes are arranged across the CPU(s) in such a way that you minimize the number of levels in the cache hierarchy you have to traverse to service accesses to shared memory.

As an illustrative example – consider a dual-socket Xeon system (which was our testing configuration). If a process makes an access to shared memory, let's be pessimistic and suppose that another process has recently accessed this memory as well, has performed a write, and that the dirty data is still in the cache hierarchy. What are the possible coherence actions that could occur? It depends on where the process was scheduled:

1. If the process was running on the same core (in a separate scheduling quantum), nothing! We have the data! Yippee! We will find it somewhere in our L1/L2 private cache hierarchy, and require no coherence actions.
2. If the process ran on another core, then on modern Xeon systems there are two ways this can pan out.
   1. The process was scheduled onto another core in our socket. This means that we will force a write-back from the other core's private cache hierarchy into the fully-shared L3, and either pull data from DRAM or directly from the other core's private cache hierarchy via the coherency fabric.
   2. The process was scheduled on the other socket. This is the worst-case, as the other socket's L3 will need to transfer data across the chipset (or we go to DRAM), at which point we can pull the data all the way through our own cache hierarchy to use it… yikes!

As we can see, there is a fairly asymmetric penalty here to service a cache miss when dealing with multi-threaded programs. So really, our goal is to minimize the number of times we have to deal with case 2.2. If we were only read-sharing across cores, everything would be hunky-dory since data is much more likely to stay cache-resident. It's really this pessimistic write-sharing behavior that starts to be annoying. Ideally, we generally prefer to trigger case 2.1 in this situation, since it balances cache miss penalties with scheduling flexibility and maximizing available compute.

The granularity of this decision tree will inevitably change depending on the CPU/configuration we are using. Zen 1 and 2, for example, would have a third branch here, as some SKUs contained separated L3 caches per CCX, so you could get away with faster access times if you managed to schedule your data-sharing processes on the same CCX.

So with some fairly strict caveats – no sub-NUMA clustering, ignoring memory residency, etc. – we set out to answer three questions:

1. How much of a benefit is there from successfully scheduling multi-threaded processes fully within one LLC domain?
2. Do current schedulers do a good job of ensuring that this happens?
3. How can they be improved to actually make this happen?

Our study mostly focused on points 1 and 2, with 3 being a "we hope this can happen", until we realized that CAS was starting to address point 3. So at this point we thought, "what if we just test CAS and EEVDF and find the gaps?". Okay, fair enough. But then we realized that we couldn't actually do that on shared lab resources and instead pivoted to profiling LAVD. It obviously exhibits very different behaviors and priorities, but the hope was that the second-order effects of LAVD would emulate that of CAS. I'm not totally sure how accurate that hypothesis ended up being, but EOTD we needed some testing methodology so this would have to do.

There's a lot more background about the internals of these schedulers about how various Linux schedulers work and our evaluation methods, but I'll fast forward to the mishaps in the next section for (relative) brevity. More fine-grained details can be found in our final report, either in PDF or web format.

So at this point, our study is clear! We want to characterize how performance is impacted by scheduling threads auto-magically with the default Linux scheduler, with a "cache-optimized" scheduler, and with manual restriction of thread positions.

As many school projects go, we started a little close to the deadline, and followed a fairly linear train of thought to start testing right away. Our experimental setup was to track coherence actions (via read-for-ownership requests sent across the socket) and to track latency figures with the various configurations, and try to drill down exactly what aspect of runtime changing had contributed to performance increases/decreases. Since our benchmarks were all run with a thread count at/under the capacity of a single socket, our proxy for an "idealized" placement across cores was to simply restrict benchmarks to a single socket via numactl as a control.

This led to one of the most striking results of our entire study: both the standard Linux scheduler and specialized LAVD scheduler perform approximately the same on our benchmarks, but using numactl to restrict thread scheduling to a single socket improved runtime by as much as 3x!

So I guess if any of you are currently using a multi-socketed machine to run workloads in an under-subscribed fashion, I would recommend at least trying to use numactl to pin workloads to a single LLC domain or a set of cores – you can gain extreme speedup in multi-threaded shared-memory programs. This is mostly due to the fact that the Linux load balancer will try to ensure that all sockets are approximately equally loaded as thread count permits, even if that actively disadvantages the host program.

The salient exception to this observation was the mediawiki benchmark, which we sourced from DCPerf – how had the runtime barely moved at all? Maybe it was the fact that cache hitrates hadn't really improved?

Okay, we can check whether that's true. Let's take a look at if the number of read-for-ownership misses (e.g L3 wants to write to something but it doesn't have the data, so it asks the other core to let it own data) has changed. When we have strong write sharing, we expect that number to go up since shared cachelines should bounce between the two sockets' L3 caches.

Whelp. Now we have two mysteries.

1. How has mediawiki not sped up at all, even with an improvement in L3 RFOs?
2. What the hell is going on with perlbench? What is making it speed up so much?

We didn't end up being able to answer either of these questions prior to the deadline unfortunately – in fact, we found this relationship the day our paper was due! So with a severe sleep deficit and copious amounts of caffeine in our bloodstreams, we wrote that we hadn't been able to pin down the issue in our final report, and continued slogging through more graphs and writing. We ended up finishing the report at exactly 11:45PM, scampered over to Siebel and slid the report under our professor's door as fast as possible. And I don't think any of us really gave it another thought after submission.

Cut to a couple weeks after end-of-semester, and I was cleaning up the traces we'd used for this project. And for some reason I have since forgotten, I was curious what the MIPS was for each of these benchmarks.

Well that makes… no sense. Throughput has gone up significantly but runtime hasn't changed?? What command did we use to run the benchmark?

 cd /home/pradyun/pg_work/dcperf/
sudo ./perf_collect_mw  "$JOBNAME" ./benchpress_cli.py run oss_performance_mediawiki_mlp  \
    -i  '{"scale_out": 1, "client_threads": 32, "duration": "10m"}' &

… Oh boy. Duration was fixed to 10 minutes. This directive was buried so deeply in our testing infrastructure, and reused so many times over the course of this project that we'd completely forgotten about it.

So in one shot, several mistakes have been made. Speedup was calculated with the assumption that instructions have been held constant, which was not the case for this benchmark. And not only have we accidentally reported another single-socket speedup as having been a phantom "no-change", we've also somehow masked the one datapoint that made LAVD look better. For the latter bit, my theory is that the nature of mediawiki suits the chain-of-tasks model that LAVD was designed for.

Side Note: LAVD was designed and optimized for the Steam Deck by Changwoo Min and the great folks over at Igalia. He had a pretty great talk about LAVD at OSPM25 which you can find here, would recommend watching!

So that was… long-winded. I hope it was at the very least entertaining! Regardless, this is one of those experiences that I hope sticks with me. For one, I hope it teaches me to stop procrastinating schoolwork so much sometimes, but also that the choice of metrics can very easily disguise results as something they're not. That can both be in the negative direction (this study) but also the positive direction. Both of these directions can be pretty dangerous, especially in environments where you fix the benchmarks and tests early on and forget about them going forwards (e.g performance engineering CI pipelines). I guess the long and short of what I'm trying to say is this:

Picking good metrics to communicate your results helps not only present findings better, but also lets your work be resilient to mistakes or oversights made in experimental design.

At a more meta-level, I feel kind of obligated to take more care with experimental methodology in future profiling studies. This was really the first time I'd profiled anything on a computer without pre-rolled tooling to do it (e.g NVIDIA's SoL tools), and it definitely showed in some of the rougher edges of our evaluation. But at the very least, I'm glad that this saga occurred within the confines of a school project rather than an actual research paper, and that I can carry the lessons here into future research.

If you're interested in some of the finer-grain details of our evaluation methodology, more performance counters from our testing, or a fairly long expository section about how LAVD and EEVDF work in Linux, I'd recommend reading our final report. It's not the most well-refined study ever, but we all learned a lot from it!

AI Disclosure: This post was written by hand and then later proofread/fact-checked with the help of an LLM. I have been an abuser of -- for longer than I can remember, and I will continue to use it – unfortunately, it gets converted into an em-dash by my Org export engine! It's truly unfortunate that hyphenation has been broadly conflated with "AI slop", yet I will continue to use it at will.

• 05/22: Client-side space overhead for Square Root ORAM was incorrectly characterized as \(O(N\log N)\)

Oblivious computation research dates back as far as 1979. Pippenger et al. [1] explored the idea of an "oblivious Turing Machine", in which the motion of a Turing Machine's head on its tape lent no information about the underlying program. That is to say, you could make no useful conclusions regarding the input program by observing the head position - its movement was indistinguishable for any two input programs of similar length. This paper not only laid out the foundations for a functional OTM, but it also was the first paper to discuss the notion of oblivious computing. Computation in which the observable accesses and movements of a machine lent no information about the underlying operations.

In their seminal 1987 papers (later reunited as a journal paper in [2]), Goldreich and Ostrovsky extended the idea of oblivious computing to an off-CPU RAM that stored some sort of software. However, their motivation had more to do with software piracy. Since programs on a RAM were still just a sequence of bits, they could easily be copied and distributed by an adversary.

The knee-jerk solution to this problem would likely be to simply encrypt the data going in and out of the RAM. However, it's important to note that where and how memory is accessed can lend important information about the underlying program, especially when paired with a high-level description of what the program does (which you get by buying the software). For example - repeated reads of a cluster of adjacent RAM blocks could indicate some sort of loop structure within the program itself. This is obviously subject to some architectural decisions on the CPU's memory subsystem, but the point should stand as a motivating example. Thus, a system allowing for oblivious memory access should be able to mask not only what the data is, but where the data is and how the data was accessed.

Today's compute landscape presents other scenarios requiring security when accessing a RAM, outside of the initial goal of piracy-prevention. A large motivating factor, as outlined in [3] by Stefanov et al., is the advent of cloud compute. Many clients store secure data (tax forms, legal documents, etc.) on remote servers that are considered "untrusted". We also heavily rely on cloud compute for many modern services, many of which will inevitably require some amount of confidential information from the user.

In a naive security model where the client and cloud use symmetric encryption to prevent data from being leaked to eavesdroppers, the access pattern can still lend information regarding the type of actions that the client is engaging in with remote storage. Furthermore, the server has knowledge of the physical memory locations that the client accesses, which inherently lends insights into the off-server actions of the client. Thus, in the early 2010s there was a renewed interest in devising secure memory schemes enabling oblivious compute with untrusted remote servers.

[2] was seminal for many reasons. Today, we recognize it as the paper that defined the security model of Oblivious RAM, and outlined the basic schemes of implementing it that motivated future ORAM schemes for nearly 2 decades.

With the above objectives, one can think of some basic implementations of ORAM. An especially trivial one would be "brute force" – regardless of the memory access, you can read every block in the memory into CPU registers, optionally modify it, re-encrypt it with a new nonce, then write it back into the same location of the RAM. If a block needs to be reused later, it can be kept in the CPU for operation even after linear scan.

This scheme isn't all bad. For one, it doesn't require any special state about block mappings on the CPU – every block is exactly where you'd expect it to be, it's just that the server doesn't know which one you actually meant to use. However, the linear scan requires \(O(N)\) memory accesses given an N-block memory, and requires \(O(N)\) bandwidth. Any practical system would rattle to a halt. This motivated early research into devising an ORAM algorithm that was bandwidth-efficient (and, as a corollary, memory-access-efficient).

As if it weren't enough to design a security definition and objective for Oblivious RAM, Goldreich and Ostrovsky outlined early optimized ORAM schemes in [2]. We discuss their initial proposal of a Square Root ORAM and its pitfalls.

Throughout the early 2000s, many ORAM schemes were devised to cut per-access bandwidth closer to this boundary. Efficient ORAM constructions like Pinkas-Reimann [5] and Goodrich-Mitzenmacher [6] still inherited from a hierarchical RAM scheme presented in section 5 of [2] and relied on oblivious sorting mechanisms. This meant that while amortized per-access ORAM latency approached logarithmic/polylogarithmic functions of N, worst-case latency was still on the order of \(\Omega(N)\) in [6] or \(\Omega(N\log N)\) in [5]. Altogether, this amounted in a performance overhead that ranged from 1400 to over 80000 times the number of operations required for the same access pattern with a transparent memory device [3] .

While some ORAM schemes around this time, like the SSS construction in [3] and Shi et al.'s \(\log^3\) ORAM [7], presented schemes that helped reduce the worst-case performance to polylogarithmic functions and reduce client/storage cost in doing so, tree-based constructions are currently the most popular category of ORAM. This is largely due to the innovations that came with the Path ORAM construction by Stefanov et al. [8].

In this section we, at a high level, discuss a variety of adjacent research endeavors that can leverage ORAM mechanisms as a facet of their system architecture or attempt to iterate on the construction of modern ORAM schemes.

First things first, I like to separate the vision of a movie from its execution. Poor execution might be the result of low budget, uncooperative cast/crew, and all other manner of factors. But a poor vision can make for a bad movie from start to finish, regardless of how well you dress it up. This is the difference between a movie like RRR and 1 Nenokkadine. RRR tends to be publicly touted as "one of India's cinematic masterpieces", but I'd argue it's a perfect example of perfect execution of a poor vision.

The best way to see that might be to imagine what RRR would be if we remove some of the "hallmark" scenes. NTR's clash with Ramcharan with all the animals and water hoses, the final fight with the two against the King, and some of the other more "extravagant" fight scenes. I feel like if you do this (which may be unfair to the film genre, by some people's opinion) then the movie immediately begins to fall flat - the plot is rather lackluster, and the movie overall attempts to justify itself by doubling down on nationalist sentiment. Overall, the vision of RRR is likely a "patriotic spectacle", and it achieves that well, but I don't know if it makes for a good movie, at least by my preference for either good plot or abundant comedy. Definitely makes for good press and eye-candy though.

By contrast, 1 Nenokkadine is a movie where some of the execution is a disservice to the vision. The plot overall is an amazing concept - a schizophrenic musician kills off his parents' murderers, only to realize he hallucinated the murders. The rest of the movie unfolds as he struggles to differentiate what parts of an elaborate conspiracy are fact or fiction. Mahesh Babu puts forward one of his career best performances in the movie, playing the tormented musician. However, the impact of the movie overall is dulled by two big things - poor pacing and plot direction in the latter half of the movie, as well as an overall lack of refinement in the way villain encounters are handled.

Some exceptions are there. The pre-interval scene shows our protagonist kill a primary antagonist thinking it to be a hallucination, only to find out he has killed the culprit in real life. Peak cinema when shown in context. But by contrast, the film is chock-full of dead space like half-baked yacht fight and bridge fight scenes. They act as a filler that don't satisfy the critic, nor the masala-loving filmgoer. The combination of the two factors above are, at least in my opinion, why the movie didn't fare well in the box office. If I were to remake a movie, 1 Nenokkadine would definitely be a top choice on my list of candidates.

As a side note, something that's been irking me about newer movies is a lack of continuous character development or screenplay. One of the bigger examples of this for me recently has been Kamal Haasan's portrayal of Vikram. He has a great performance and the fight scenes are beautiful - the main reasons why the movie fared so well - but his motivation as a character effectively unravels towards the end. We witness the narrative from the perspective of a man attempting to avenge the unjust murder of his son, only to encounter a climax monologue about a drug-free society and no mention of the dead son. The disconnect, especially since it was towards the end, left a sour taste in my mouth, and gave me a negative final opinion on the movie.

Vidaamuyarchi suffered from an opposite problem - it spends about 40 minutes on exposition of why a husband and wife met, married, grew apart, then divorced. This time is all spent incessantly cutting between different flashbacks, in a seemingly random order. I feel that giving these flashbacks in a more orderly manner - for example, recounting the circumstances of their relationship to a divorce lawyer as a conversation - would have made for a much more linear narrative, cutting this exposition to only 25 or 30 minutes. The remaining time could have been used to pace the final stages of the movie better, which felt extremely rushed in the chosen screenplay.

This leads me to believe that the writing and screenplay for movies has been weakening, while the idea and plots still have some originality or quality to them. Though that's just my opinion as a tired amateur - I'm sure experts have done much more in-depth analysis of these things.