Adam J King

Pull up a seat with FS2 Streams

Thu, 26 Feb 2026 00:00:00 +0000

I usually put FS2 amongst my top favourite libraries in my career. This streaming library powers some of the most popular Scala Typelevel libraries. I often encounter streaming design problems that make me think to myself, “you can do this with a couple lines of FS2”. Enough gushing. I typically find that folks who come across FS2 struggle with creating a correct mental model for how a stream in FS2 works.

This article assumes you know about Cats-Effect and FS2 already and would like to dive deeper. Understanding how FS2 works requires a mental model for how suspended side effects work too.

On the surface, an FS2 Stream looks and feels a lot like a collection (like List). It supports common collections operations like map, flatMap, or filter. In fact, when we go to complete a stream, we can convert our stream to a host of different collection types.

1
2
3
4


val stream: Stream[F, Int] = ???

stream.compile.toList // F[List[Int]]
stream.compile.toVector // F[Vector[Int]]

This property of Streams makes it easy to understand the API at first, but can make it difficult to understand how it works internally. A Stream differs from a List in that it incorporates an effect type (F[_]) as part of its definition (Stream[F, A]). So why do we need an effect type for our stream definition? A Stream does not necesserily hold all elements in memory at once. For example, if we stream an HTTP response, not all body bytes arrive from the network at the same time.

When using an effectful type (like IO), a Stream[IO, A] actually represents a single IO[Option[A]] that we call repeatedly for new results. We refer to this as a “pull-based” stream because we need to perform an action to retrieve the next elements of the stream. This works the same as Scala’s Iterator class when lazy generating results. As this diagram shows, we don’t materialise the elements of the Stream until we reach them.

Another important thing to note about Streams; they don’t (always) contain single elements. Instead, they contain multiple groups of elements (Chunks), which allows individual pulls to return more than one element. Useful when working with things like response payloads where consuming bodies byte by byte could harm performance. At one time a Stream only holds a single chunk. It pulls a new chunk as necessary when we consume the stream.

Pull API

Armed with this understanding of how a Stream works we can actually access the underlying representation type, Pull. Given an arbitrary stream we can call .pull to convert a Stream into a Pull. This type covers some internal implementation details but, in a nutshell, a Pull monad represents a single “step” of the underlying representation. For example, a single Pull might represent polling a connection for new response bytes.

Technically .pull returns a ToPull type. Again, this type covers some internal implementation details but acts as a springboard for calls like uncons or peek, which can optionally consume the current chunk. Operations on this type always yield a Pull so for our purposes I will focus on that.

A Pull type can cause confusion because it has three type parameters.

1
2
3
4
5
6
7


 Pull[F, O, R]
 ▲ ▲ ▲
 │ │ └─────── Return Type
 │ │
 │ └────────── Output Type
 │
 └───────────── Effect Type

The effect type simply represents the monad in use (e.g. IO). The output type represents the type of the parent Stream. The return type represents a “working” type to allow use to compose multiple Pulls into a single stream step. We can only collapse Pulls back to a normal Stream if they return a Unit type (signalling they are “done”), using the Pull.stream API. This distinction can make Pull a confusing type to work with at first.

1
2
3
4
5
6
7
8



val pull: Pull[IO, String, Int] =
 for {
 i <- Pull.pure(1)
 _ <- Pull.output1(i.toString)
 } yield i + 1

val stream: Stream[IO, String] = pull >> Pull.done

Pull.output1 emits the given value as an element in our stream. Pull.done just represents a Pull[F, Nothing, Unit] and acts as a convenient way of signalling the current pull has finished outputting any elements.

You can define custom steps on an existing Stream by using .repeatPull. This gives you a high degree of control over when and how we pull elements from the upstream. This function provides a base ToPull and asks you to return an Option[Stream[F, A]] to indicate if and how the stream should continue. An interesting property of this stage is that we don’t have to return elements from our upstream definition. We can insert new elements or switch to a different stream entirely.

Motivation and Summary

FS2’s Stream already provides a lot of powerful abstractions for working on a stream without dipping into the pull API, so why might you want to?

Performant consumption of chunks. Individually stream operations can provide either full chunks (via .chunks) or single elements. If you ever want to combine elements across chunks or dynamically merge chunks then you will need to pull and merge chunks yourself.
Single step logic. Sometimes you might want to partially consume a stream before making a decision on how to process the full stream. The pull API allows you to arbitrarily consume the Stream without committing to a single strategy.
Full control over execution. You might have a sensitive upstream API where you want to carefully choose when and how effects get executed.

Understanding how a Stream works underneath can help you intuit what a stream can (and can’t) do, as well as diagnosing any performance issues. The pull API unlocks performance improvements around chunk handling that the typical Stream interface might not provide. This might seem like a dense topic at first, but sometimes the best way to learn something means using it. Then you should find the concepts click into place (and if they don’t, I apologise unreservedly).

On the Dangers of Saying Things

Thu, 22 Jan 2026 00:00:00 +0000

I struggle when it comes to writing something publicly. The things you say reflect on you as an individual, but usually my reluctance comes from my belief that what I have to say carries little importance. Publishing something publicly requires the belief that you have something worth sharing. You need to keep believing this even if no one hears it at all! This strong level of conviction to scream into the void doesn’t come naturally. Few people possess that level of self-assurance, and rarely do people who think they have something worth listening to also care about hearing something back.

So why say anything at all? It does have an air of hypocrisy to it, complaining about the self-serving nature of blogging. Ultimately, I think the act of writing in this form has some benefits. Life doesn’t need permission to exist any more than we need permission to have an opinion. Our views and collected knowledge only strengthen from writing them down, and publishing them creates pressure to refine and improve our ideas. I’m not trying to do anything more here than say, “Hey, I exist, and maybe what I have to say makes you feel like you exist too”.

I am the target reader of this post. I would like to write posts that me from another time might like to read. If you like them too, then by all means, keep reading.

21st Century Baby Brain

Fri, 26 Sep 2025 00:00:00 +0000

I recently became a Dad (for the first time). We don’t want to share photos online, but trust me — this baby looks great. Beautiful blue eyes matched with amber hair, and a screaming cry that bursts both eardrums! While on paternity leave for the last two months we’ve had to overcome a lot of challenges. A lot of these make me reflect on the way I operate as a software engineer of nearly a decade now, or even as a human-being of thirty years.

No such thing as good advice

The art of raising a child comes with a lot of advice. Some of it scientifically based, others from years of experience, and some just plain misguided or wrong. In truth these unique little creatures have many needs, and you can’t always apply the right advice at the right time. While they may share a lot of themes each newborn still has a unique experience. Our ability to read these cues (or miss them completely) usually becomes the cause of a lot of our day-to-day frustrations.

When we meet challenges in our work projects we try to fall back on the rules we know. As developers, we have a special relationship with rules because we spend most of our days searching for and implementing them. Best practices can even become our identities as we spend our careers advocating for them. When those rules don’t work we get frustrated because things should get better, because we did everything “right”. In those situations people often double down, we conclude that we just didn’t implement our rules properly. In truth the rules don’t always work. Good advice can fail in the wrong moment, or it solves the wrong problem.

In a lot of situations our little one, and us, find ourselves frustrated because we don’t understand each other. Programming errors always have a cause (even when they come from unexpected places). You can debug a program to get to the root of the problem, but sadly you can’t debug a newborn baby. You can’t debug a human adult either. Despite this we still try to apply our rules to everyday interactions in a work project. We want to codify our processes because it makes us feel better — even when the Agile manifesto tells us to put “people over processes”. Instead, if we listened to the right cues we might avoid a lot of disagreements over the best way to approach a project.

You can’t spoil a baby

Something you should hear a lot as a new parent — “You can’t spoil a baby”. Newborn babies need to know you hear their cries and will appear when you need them. In those early days they need as much reassurance and love as possible. This stays true for their childhood, teenage years, and beyond. Children need to know that you understand and accept their feelings, even if you don’t agree with them. This allows them to build better emotional understanding and therefore tools to manage their emotions later in life. I think this stays true for adults too, we need to approach conflict in life with love and understanding. Everyone in the room should want to achieve the best outcome and come together in good faith. By showing kindness to each other we make it so that people enjoy working with us, and in turn has the desire do good work.

No walking away

This kind of empathy takes an enormous amount of effort to pull off sometimes. Not every tantrum from a baby feels justified, and in a sleep-deprived state it stings even more. When a programming problem gets us down we can usually close our laptops and walk away. You can’t say the same for a newborn baby, the task at hand still needs doing. We need to find the will to push through and achieve our goal, or find a compromise that works for the situation. Sometimes for me that means walking up and down the stairs until she falls asleep.

The software we build doesn’t handle life and death most of the time. But it can handle high stakes decisions; payments taken, payments not taken, forms submitted, or deadlines missed. I think the best software design stays flexible. It should cope with all states (illegal or not), all inputs, and where possible all outputs too. We often throw exceptions or halt our execution flow when things go wrong. Instead, we could return sensible defaults, or define our state model with more flexibility.

Changing worlds

These days some people write their content with ChatGPT, I wrote this with a baby on my lap. She uses fewer emojis than ChatGPT and hasn’t made a single useful suggestion yet! These AI agents have only just exploded into our lives, but for her, they will seem normal and common-place. In my lifetime we started with basic brick phones, before touchscreen smartphones became the new thing, and eventually became old too. Things that feel new and exciting to us now might pale compared to what comes next — something equally exciting and daunting to think about. Life will always keep changing around us, and I look forward to seeing it all through her eyes.

Right now, Edith the baby just learned to smile. She loves to stand (with Dad’s support) while smiling and dribbling everywhere. I return to the working world with a little resentment and a little excitement.

Enjoy The View

Tue, 20 Aug 2024 00:00:00 +0000

Aimed at relatively new Scala developers or those unfamiliar with Scala collection performance outside the usual List or Seq types.

Most of the common collection types in Scala are implemented strictly; this means that at all times every element of the collection is calculated and stored in memory. This can have performance impacts if our collections come from expensive operations, contain a high number of objects, or contain particularly large objects. Some methods such as find or head don’t need to read every element of the collection and so using strict collections isn’t always necessary. A good example of a collection that benefits from being lazy is the linked list (Scala’s List type). Linked lists are implemented as an element and a pointer to the next element, instead we can make the collection lazy by storing a function for the next pointer instead. See the example below for how this works.

1
2


strictList = 1 -> 2 -> 3 -> 4 -> 5
lazyList = 1 -> (n => n + 1)

Lists in Scala are not lazy by default and so if we want this behaviour we’ll actually need to use List’s sister class LazyList. This type works a lot like a list but instead the list is stored as an element and a “thunk”, a function stored on the heap which can be used to calculate the next element. Now when we use our find or head operations we only calculate as many elements as we need rather than creating elements unnecessarily. It also opens up some interesting algorithm designs that make use of infinite lists. For example, implementing a zipWithIndex function would look like this.

1
2


scala> LazyList.range(0, Int.MaxValue).zip("a cool example").toList
val res0: List[(Int, Char)] = List((0,a), (1, ), (2,c), (3,o), (4,o), (5,l), (6, ), (7,e), (8,x), (9,a), (10,m), (11,p), (12,l), (13,e))

Notice how the initial LazyList.range(0, Int.MaxValue) goes all the way to the maximum integer. If you replaced the LazyList with a regular strict List you’ll find the program takes a lot longer to complete (if it even does at all).

Lazy lists actually retain their elements after they’re calculated so an infinitely sized LazyList can still risk overrunning your heap memory! If you want to discard elements you’ll need to work recursively (with tail call optimization)

What if we’re not working with a lazy collection? We can actually work with any collection lazily by using a View. You can summon a view for a collection by using .view and continue to use the familiar collection traversal methods. The upside is that every operation we do will be evaluated lazily just like the lazy list!

This example shows how the order of operations differs between a strict list and a lazy view on the same collection.

1
2
3
4
5
6
7
8
9


val normalList = List.range(0, 2)
val viewedList = normalList.view

val tappedList = normalList.tapEach(_ => println("Operation A")).tapEach(_ => println("Operation B"))
val tappedView = viewedList.tapEach(_ => println("Operation A")).tapEach(_ => println("Operation B"))

println("Not done yet?")
// realise the view
tappedView.toList

Run this snippet and you’ll see something interesting.

1
2
3
4
5
6
7
8
9


Operation A
Operation A
Operation B
Operation B
Not done yet?
Operation A
Operation B
Operation A
Operation B

The strict list evaluates the entire list first for the first tapEach, and then evaluates it again for the second tapEach (meaning we’ve actually crossed the whole list twice). In the view however the operations are actually combined making our update more efficient. Similarly we actually don’t see the output of the view until we force it back into a list later on (sometimes referred to as “realising” the list). A view isn’t always the right choice for the job. Views are great for traversal but don’t support functions that might access the collection in a random order (for example, sorting). Storing unevaluated thunks can also sometimes be more costly then simply evaluating the collection to begin with, for example situations where we might build a lot of views in one go.

Turning Docker Compositions into Test Suites

Wed, 06 Sep 2023 00:00:00 +0000

Use Docker compositions in tests with Test-Containers and ScalaTest.

Sample code repository.

We use docker-compose as a tool that can build, deploy, and manage multiple containers on the same network. These configurations, known as compositions, remove a lot of the manual setup we often associate with running infrastructure locally. It provides local versions of dependencies for a faster, more realistic feedback loop when developing applications.

Our application level tests could look as simple as this;

Test Containers

For our tests we will use test-containers. It offers us;

Access to a Docker context from within tests.
Integration with test harnesses provided by popular testing frameworks.
The automatic creation and clean-up of Docker containers.

Test-Containers has a Docker-Compose plugin that allows us to run compositions from our tests. First, let’s define a simple composition file.

I’ve kept it simple by only including our application, but from this point on I can extend it with whatever setup we need without having to change our tests at all. To start using this in our tests we’ll need to do three things.

Build and publish our application image locally.
Integrate our composition setup into our test-suite.
Create a client that can call our application.

1. Build and publish our application image locally.

Before we can use it in a test suite we need to build our application into a Docker image. Unfortunately Docker-Compose does not have a concept of running external build tasks to get an image. That said most build tools will allow us to build an image before running our test suite. With SBT and native-packager plugin makes it easy to add our Docker build stage as a prerequisite for our integration tests.

An SBT quirk means we need to specify our Docker build step for every test task.

1
2
3


 Test / test := (Test / test).dependsOn(server / Docker / publishLocal).value,
 Test / testOnly := (Test / testOnly).dependsOn(server / Docker / publishLocal).evaluated,
 Test / testQuick := (Test / testQuick).dependsOn(server / Docker / publishLocal).evaluated

2. Integrate our composition setup into our test-suite.

Next, we add the image details to our tests.

The Test-Containers library has integrations for many test frameworks, which makes this easier, but we’ll still need to do some wiring. The Docker-Compose module needs to know the location of our compose-file. You might want to hard-code this but that affects the portability of your tests. Instead, I recommend passing the location in as a property and using SBT to locate the file. This has the benefits us by enabling testing against multiple compose-files; useful for testing against different environments or setups.

If you use an IDE to run test-suites you may need to update the runner config before this will work. In IntelliJ, you can save a run configuration and share it by committing it to the repo.

We need to wait for our containers to start before using them. We can get around this by making our access details lazy or use the Test-Containers helpers (which run after the containers finish starting).

Lastly we need to instruct Test-Containers on how to use our containers.

Some additional considerations you might make;

We added a wait condition for the service’s health-check so that Test-Containers knows to start testing. For more complex start conditions check out the test-containers documentation.
We only listen to logs of the main container as logging everything at once creates too much noise. To help with that we can either:
- Set noisy application’s log-levels through the Docker environment.
- Use Test-Containers logging classes to build something custom.

3. Create a client to call our application.

Finally let’s create our HTTP client to call our service. We have full access to the Docker containers and, if needed, we can connect to our dependencies directly. Alternatively if you provide clients for your service you can use them here instead.

Summary

With our test fixture code in place we can now start writing code without too much concern about the state of our containers.

Now if we hit a test failure we can launch our docker-composition to debug our test case. This helps us switch to an iterative, live feedback loop for manual testing and experimentation.

Custom Request DSLs with Http4s

Thu, 30 Mar 2023 00:00:00 +0000

How Http4s leverages Scala’s powerful pattern-matching features, and how you can use them too.

I’ve recently played around with Http4s’s routing DSL and added some custom mechanics of my own. Sometimes we want to access request information generically independent if the route’s normal purpose. Http4s' AuthedRoutes serves as a great example for this.

1
2
3
4
5
6
7


val normal: HttpRoutes[IO] = HttpRoutes.of {
 case GET -> / "user" / UserId(user) / "resource"
}

val authed: AuthedRoutes[UserId, IO] = AuthedRoutes.of {
 case GET -> / "user" / "resource" as user
}

The second route shows an example of using some new DSL syntax. That small as user at the end of the second route definition does a lot of heavy lifting around user authentication. Even better as it modifies the route type itself we can’t forget to add authentication to a route (which we might if added auth route by route).

For my change I will focus on a relatively neutral change that works with the normal Http4s request and response types. Let’s work with a simple requirement.

Modify the route’s behaviour based on the User-Agent of the caller.

How might that look? Let’s sketch out a possible DSL.

1
2
3


HttpRoutes.of[IO] {
 case Service.Distributor using GET -> Root / "resource" / IntVar(id) =>
}

To understand how we might add this we need to explore how Http4s cleverly uses Scala’s pattern match system. When we pattern match on a case class Scala invokes a method called unapply to decide if the given value “matches”. Typically, that method might look something like this.

1
2
3


object Foo {
 def unapply(foo: Foo): Option[Int]
}

In this case the implementation knows how to decompose a Foo into a Int, by using the class’ foo field. Sometimes we can’t access a field like that (for example, an ADT) and so the optional return type allows us to indicate the match failed. This method doesn’t have to be synthetic and Scala allows us to define our own custom unapply matchers, which Scala refers to as extractor objects.

Http4s itself has some great examples of when we might want to use extractors. The status matcher Succesful matches response statuses within the 2xx range, with similar matchers for 4xx and 5xx too. This showcases the first style of matcher — a way of grouping similar terms in a match.

You can nest Scala’s matchers in the same way that you can nest data. For example;

1
2
3
4
5


 case Right(Some(3)) =>
 // ^ ^
 // | |_ Then it applies the `Option` matcher to the result.
 // |
 // |_ First applies the `Either` matcher

This nesting also powers the familiar list matcher.

1
2
3


 case 1 :: 2 :: Nil =>
 // really looks like
 case ::(1, ::(2, ::(3, ::(Nil))) =>

This syntax eliminates the normal bracket syntax see in favour of something more readable. Seeing this you might see how the Http4s DSL takes shape. Using DSL objects (made available via the Htp4sDSL) trait we can string a series of matchers along to define our expected request structure. Checking the source code gives us an idea on how to structure our own custom matcher.

The first component of a Http4s route:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


object -> {

 /** HttpMethod extractor:
 * {{{
 * (request.method, Path(request.path)) match {
 * case Method.GET -> Root / "test.json" => ...
 * }}}
 */
 def unapply[F[_]](req: Request[F]): Some[(Method, Path)] = ...
}

It takes as input the inbound request itself and outputs two chunks split from the request; the method and the path. Through the same syntax as :: that we saw earlier; Scala can match the request as a method on the left and a path on the right. Scala also allows us to match against specific values, so we can further refine our match with an exact method. The path however gets fed to another Path matcher, which decomposes the value further.

1
2
3


object / {
 def unapply(path: Path): Option[(Path, String)] = ...
}

This path matcher consumes a Path and splits to a String component on its right-hand side. The left-hand side returns another Path so we can decompose the path string as much as we need to.

You should see a pattern emerging, and we can start implementing our own. Let’s revisit our desired syntax.

1
2
3


HttpRoutes.of[IO] {
 case Service.Distributor using GET -> Root / "resource" / IntVar(id) =>
}

We can start building our matcher by looking at our preferred inputs and outputs. In this case we will need:

1
2


Input: Request[IO]
Output: (Service , Request[IO])

It takes a request and returns it plus a newly created Service object. Returning the request enables us to continue matching on it down the line. A trivial implementation looks a little like this;

Notice the unapply method actually has a generic type parameter. Pattern matches do not support type parameters so if you include them they must be inferred from the matcher input.

The logic here could become more complex if we wanted it to, for example we could extract a specific user-agent version too. Like AuthedRoutes the best syntax additions are obvious and limited. By changing the order of the outputs we could move our syntax the end of the match rather than the front. Experimenting with the values you extract in your match brings up some interesting possibilities. Take Http4s’ ->> matcher as an example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


 /** Extractor to match an http resource and then enumerate all supported methods:
 * {{{
 * (request.method, Path(request.path)) match {
 * case withMethod ->> Root / "test.json" => withMethod {
 * case Method.GET => ...
 * case Method.POST => ...
 * }}}
 *
 * Returns an error response if the method is not matched, in accordance with [[https://datatracker.ietf.org/doc/html/rfc7231#section-4.1 RFC7231]]
 */
 def unapply[F[_]: Applicative](
 req: Request[F]
 ): Some[(PartialFunction[Method, F[Response[F]]] => F[Response[F]], Path)] = ...

Returning a partial-function yields control back to the caller, who can add their own custom behaviour. This allows for some powerful DSLs that can make defining request-dependent service behaviour a breeze.

The big question is… should you? This technique can help simplify a lot of code, but it can make it worse too. Common drawbacks and criticisms of custom DSLs include;

Confusing or pointlessly obtuse syntax (for example, |@| from cats).
They can hide too much. Subtle behaviours introduced through syntax can feel “magic”.
Newcomers need to learn your syntax on top of a language they might not know already.

You and your team decide how far you want to take it. A part of the art of software development includes deciding when to use these kinds of techniques.

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” ― Antoine de Saint-Exupéry

Even small uses can benefit from custom matchers, for example this…

1

 case Left(Some(event)) if event.timestamp > cutOffTime =>

…into this…

1

 case LiveEvent(event) =>

Which reduces cramped matchers into something simple and self-documenting.

Links

Mon, 01 Jan 0001 00:00:00 +0000

To use this feature, add links section to frontmatter.

This page’s frontmatter:

1
2
3
4
5
6
7
8
9


links:
 - title: GitHub
 description: GitHub is the world's largest software development platform.
 website: https://github.com
 image: https://github.githubassets.com/images/modules/logos_page/GitHub-Mark.png
 - title: TypeScript
 description: TypeScript is a typed superset of JavaScript that compiles to plain JavaScript.
 website: https://www.typescriptlang.org
 image: ts-logo-128.jpg

image field accepts both local and external images.

Search