Programming on Adam J King

Pull up a seat with FS2 Streams

Thu, 26 Feb 2026 00:00:00 +0000

I usually put FS2 amongst my top favourite libraries in my career. This streaming library powers some of the most popular Scala Typelevel libraries. I often encounter streaming design problems that make me think to myself, “you can do this with a couple lines of FS2”. Enough gushing. I typically find that folks who come across FS2 struggle with creating a correct mental model for how a stream in FS2 works.

This article assumes you know about Cats-Effect and FS2 already and would like to dive deeper. Understanding how FS2 works requires a mental model for how suspended side effects work too.

On the surface, an FS2 Stream looks and feels a lot like a collection (like List). It supports common collections operations like map, flatMap, or filter. In fact, when we go to complete a stream, we can convert our stream to a host of different collection types.

1
2
3
4


val stream: Stream[F, Int] = ???

stream.compile.toList // F[List[Int]]
stream.compile.toVector // F[Vector[Int]]

This property of Streams makes it easy to understand the API at first, but can make it difficult to understand how it works internally. A Stream differs from a List in that it incorporates an effect type (F[_]) as part of its definition (Stream[F, A]). So why do we need an effect type for our stream definition? A Stream does not necesserily hold all elements in memory at once. For example, if we stream an HTTP response, not all body bytes arrive from the network at the same time.

When using an effectful type (like IO), a Stream[IO, A] actually represents a single IO[Option[A]] that we call repeatedly for new results. We refer to this as a “pull-based” stream because we need to perform an action to retrieve the next elements of the stream. This works the same as Scala’s Iterator class when lazy generating results. As this diagram shows, we don’t materialise the elements of the Stream until we reach them.

Another important thing to note about Streams; they don’t (always) contain single elements. Instead, they contain multiple groups of elements (Chunks), which allows individual pulls to return more than one element. Useful when working with things like response payloads where consuming bodies byte by byte could harm performance. At one time a Stream only holds a single chunk. It pulls a new chunk as necessary when we consume the stream.

Pull API

Armed with this understanding of how a Stream works we can actually access the underlying representation type, Pull. Given an arbitrary stream we can call .pull to convert a Stream into a Pull. This type covers some internal implementation details but, in a nutshell, a Pull monad represents a single “step” of the underlying representation. For example, a single Pull might represent polling a connection for new response bytes.

Technically .pull returns a ToPull type. Again, this type covers some internal implementation details but acts as a springboard for calls like uncons or peek, which can optionally consume the current chunk. Operations on this type always yield a Pull so for our purposes I will focus on that.

A Pull type can cause confusion because it has three type parameters.

1
2
3
4
5
6
7


 Pull[F, O, R]
 ▲ ▲ ▲
 │ │ └─────── Return Type
 │ │
 │ └────────── Output Type
 │
 └───────────── Effect Type

The effect type simply represents the monad in use (e.g. IO). The output type represents the type of the parent Stream. The return type represents a “working” type to allow use to compose multiple Pulls into a single stream step. We can only collapse Pulls back to a normal Stream if they return a Unit type (signalling they are “done”), using the Pull.stream API. This distinction can make Pull a confusing type to work with at first.

1
2
3
4
5
6
7
8



val pull: Pull[IO, String, Int] =
 for {
 i <- Pull.pure(1)
 _ <- Pull.output1(i.toString)
 } yield i + 1

val stream: Stream[IO, String] = pull >> Pull.done

Pull.output1 emits the given value as an element in our stream. Pull.done just represents a Pull[F, Nothing, Unit] and acts as a convenient way of signalling the current pull has finished outputting any elements.

You can define custom steps on an existing Stream by using .repeatPull. This gives you a high degree of control over when and how we pull elements from the upstream. This function provides a base ToPull and asks you to return an Option[Stream[F, A]] to indicate if and how the stream should continue. An interesting property of this stage is that we don’t have to return elements from our upstream definition. We can insert new elements or switch to a different stream entirely.

Motivation and Summary

FS2’s Stream already provides a lot of powerful abstractions for working on a stream without dipping into the pull API, so why might you want to?

Performant consumption of chunks. Individually stream operations can provide either full chunks (via .chunks) or single elements. If you ever want to combine elements across chunks or dynamically merge chunks then you will need to pull and merge chunks yourself.
Single step logic. Sometimes you might want to partially consume a stream before making a decision on how to process the full stream. The pull API allows you to arbitrarily consume the Stream without committing to a single strategy.
Full control over execution. You might have a sensitive upstream API where you want to carefully choose when and how effects get executed.

Understanding how a Stream works underneath can help you intuit what a stream can (and can’t) do, as well as diagnosing any performance issues. The pull API unlocks performance improvements around chunk handling that the typical Stream interface might not provide. This might seem like a dense topic at first, but sometimes the best way to learn something means using it. Then you should find the concepts click into place (and if they don’t, I apologise unreservedly).

Enjoy The View

Tue, 20 Aug 2024 00:00:00 +0000

Aimed at relatively new Scala developers or those unfamiliar with Scala collection performance outside the usual List or Seq types.

Most of the common collection types in Scala are implemented strictly; this means that at all times every element of the collection is calculated and stored in memory. This can have performance impacts if our collections come from expensive operations, contain a high number of objects, or contain particularly large objects. Some methods such as find or head don’t need to read every element of the collection and so using strict collections isn’t always necessary. A good example of a collection that benefits from being lazy is the linked list (Scala’s List type). Linked lists are implemented as an element and a pointer to the next element, instead we can make the collection lazy by storing a function for the next pointer instead. See the example below for how this works.

1
2


strictList = 1 -> 2 -> 3 -> 4 -> 5
lazyList = 1 -> (n => n + 1)

Lists in Scala are not lazy by default and so if we want this behaviour we’ll actually need to use List’s sister class LazyList. This type works a lot like a list but instead the list is stored as an element and a “thunk”, a function stored on the heap which can be used to calculate the next element. Now when we use our find or head operations we only calculate as many elements as we need rather than creating elements unnecessarily. It also opens up some interesting algorithm designs that make use of infinite lists. For example, implementing a zipWithIndex function would look like this.

1
2


scala> LazyList.range(0, Int.MaxValue).zip("a cool example").toList
val res0: List[(Int, Char)] = List((0,a), (1, ), (2,c), (3,o), (4,o), (5,l), (6, ), (7,e), (8,x), (9,a), (10,m), (11,p), (12,l), (13,e))

Notice how the initial LazyList.range(0, Int.MaxValue) goes all the way to the maximum integer. If you replaced the LazyList with a regular strict List you’ll find the program takes a lot longer to complete (if it even does at all).

Lazy lists actually retain their elements after they’re calculated so an infinitely sized LazyList can still risk overrunning your heap memory! If you want to discard elements you’ll need to work recursively (with tail call optimization)

What if we’re not working with a lazy collection? We can actually work with any collection lazily by using a View. You can summon a view for a collection by using .view and continue to use the familiar collection traversal methods. The upside is that every operation we do will be evaluated lazily just like the lazy list!

This example shows how the order of operations differs between a strict list and a lazy view on the same collection.

1
2
3
4
5
6
7
8
9


val normalList = List.range(0, 2)
val viewedList = normalList.view

val tappedList = normalList.tapEach(_ => println("Operation A")).tapEach(_ => println("Operation B"))
val tappedView = viewedList.tapEach(_ => println("Operation A")).tapEach(_ => println("Operation B"))

println("Not done yet?")
// realise the view
tappedView.toList

Run this snippet and you’ll see something interesting.

1
2
3
4
5
6
7
8
9


Operation A
Operation A
Operation B
Operation B
Not done yet?
Operation A
Operation B
Operation A
Operation B

The strict list evaluates the entire list first for the first tapEach, and then evaluates it again for the second tapEach (meaning we’ve actually crossed the whole list twice). In the view however the operations are actually combined making our update more efficient. Similarly we actually don’t see the output of the view until we force it back into a list later on (sometimes referred to as “realising” the list). A view isn’t always the right choice for the job. Views are great for traversal but don’t support functions that might access the collection in a random order (for example, sorting). Storing unevaluated thunks can also sometimes be more costly then simply evaluating the collection to begin with, for example situations where we might build a lot of views in one go.

Turning Docker Compositions into Test Suites

Wed, 06 Sep 2023 00:00:00 +0000

Use Docker compositions in tests with Test-Containers and ScalaTest.

Sample code repository.

We use docker-compose as a tool that can build, deploy, and manage multiple containers on the same network. These configurations, known as compositions, remove a lot of the manual setup we often associate with running infrastructure locally. It provides local versions of dependencies for a faster, more realistic feedback loop when developing applications.

Our application level tests could look as simple as this;

Test Containers

For our tests we will use test-containers. It offers us;

Access to a Docker context from within tests.
Integration with test harnesses provided by popular testing frameworks.
The automatic creation and clean-up of Docker containers.

Test-Containers has a Docker-Compose plugin that allows us to run compositions from our tests. First, let’s define a simple composition file.

I’ve kept it simple by only including our application, but from this point on I can extend it with whatever setup we need without having to change our tests at all. To start using this in our tests we’ll need to do three things.

Build and publish our application image locally.
Integrate our composition setup into our test-suite.
Create a client that can call our application.

1. Build and publish our application image locally.

Before we can use it in a test suite we need to build our application into a Docker image. Unfortunately Docker-Compose does not have a concept of running external build tasks to get an image. That said most build tools will allow us to build an image before running our test suite. With SBT and native-packager plugin makes it easy to add our Docker build stage as a prerequisite for our integration tests.

An SBT quirk means we need to specify our Docker build step for every test task.

1
2
3


 Test / test := (Test / test).dependsOn(server / Docker / publishLocal).value,
 Test / testOnly := (Test / testOnly).dependsOn(server / Docker / publishLocal).evaluated,
 Test / testQuick := (Test / testQuick).dependsOn(server / Docker / publishLocal).evaluated

2. Integrate our composition setup into our test-suite.

Next, we add the image details to our tests.

The Test-Containers library has integrations for many test frameworks, which makes this easier, but we’ll still need to do some wiring. The Docker-Compose module needs to know the location of our compose-file. You might want to hard-code this but that affects the portability of your tests. Instead, I recommend passing the location in as a property and using SBT to locate the file. This has the benefits us by enabling testing against multiple compose-files; useful for testing against different environments or setups.

If you use an IDE to run test-suites you may need to update the runner config before this will work. In IntelliJ, you can save a run configuration and share it by committing it to the repo.

We need to wait for our containers to start before using them. We can get around this by making our access details lazy or use the Test-Containers helpers (which run after the containers finish starting).

Lastly we need to instruct Test-Containers on how to use our containers.

Some additional considerations you might make;

We added a wait condition for the service’s health-check so that Test-Containers knows to start testing. For more complex start conditions check out the test-containers documentation.
We only listen to logs of the main container as logging everything at once creates too much noise. To help with that we can either:
- Set noisy application’s log-levels through the Docker environment.
- Use Test-Containers logging classes to build something custom.

3. Create a client to call our application.

Finally let’s create our HTTP client to call our service. We have full access to the Docker containers and, if needed, we can connect to our dependencies directly. Alternatively if you provide clients for your service you can use them here instead.

Summary

With our test fixture code in place we can now start writing code without too much concern about the state of our containers.

Now if we hit a test failure we can launch our docker-composition to debug our test case. This helps us switch to an iterative, live feedback loop for manual testing and experimentation.

Custom Request DSLs with Http4s

Thu, 30 Mar 2023 00:00:00 +0000

How Http4s leverages Scala’s powerful pattern-matching features, and how you can use them too.

I’ve recently played around with Http4s’s routing DSL and added some custom mechanics of my own. Sometimes we want to access request information generically independent if the route’s normal purpose. Http4s' AuthedRoutes serves as a great example for this.

1
2
3
4
5
6
7


val normal: HttpRoutes[IO] = HttpRoutes.of {
 case GET -> / "user" / UserId(user) / "resource"
}

val authed: AuthedRoutes[UserId, IO] = AuthedRoutes.of {
 case GET -> / "user" / "resource" as user
}

The second route shows an example of using some new DSL syntax. That small as user at the end of the second route definition does a lot of heavy lifting around user authentication. Even better as it modifies the route type itself we can’t forget to add authentication to a route (which we might if added auth route by route).

For my change I will focus on a relatively neutral change that works with the normal Http4s request and response types. Let’s work with a simple requirement.

Modify the route’s behaviour based on the User-Agent of the caller.

How might that look? Let’s sketch out a possible DSL.

1
2
3


HttpRoutes.of[IO] {
 case Service.Distributor using GET -> Root / "resource" / IntVar(id) =>
}

To understand how we might add this we need to explore how Http4s cleverly uses Scala’s pattern match system. When we pattern match on a case class Scala invokes a method called unapply to decide if the given value “matches”. Typically, that method might look something like this.

1
2
3


object Foo {
 def unapply(foo: Foo): Option[Int]
}

In this case the implementation knows how to decompose a Foo into a Int, by using the class’ foo field. Sometimes we can’t access a field like that (for example, an ADT) and so the optional return type allows us to indicate the match failed. This method doesn’t have to be synthetic and Scala allows us to define our own custom unapply matchers, which Scala refers to as extractor objects.

Http4s itself has some great examples of when we might want to use extractors. The status matcher Succesful matches response statuses within the 2xx range, with similar matchers for 4xx and 5xx too. This showcases the first style of matcher — a way of grouping similar terms in a match.

You can nest Scala’s matchers in the same way that you can nest data. For example;

1
2
3
4
5


 case Right(Some(3)) =>
 // ^ ^
 // | |_ Then it applies the `Option` matcher to the result.
 // |
 // |_ First applies the `Either` matcher

This nesting also powers the familiar list matcher.

1
2
3


 case 1 :: 2 :: Nil =>
 // really looks like
 case ::(1, ::(2, ::(3, ::(Nil))) =>

This syntax eliminates the normal bracket syntax see in favour of something more readable. Seeing this you might see how the Http4s DSL takes shape. Using DSL objects (made available via the Htp4sDSL) trait we can string a series of matchers along to define our expected request structure. Checking the source code gives us an idea on how to structure our own custom matcher.

The first component of a Http4s route:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


object -> {

 /** HttpMethod extractor:
 * {{{
 * (request.method, Path(request.path)) match {
 * case Method.GET -> Root / "test.json" => ...
 * }}}
 */
 def unapply[F[_]](req: Request[F]): Some[(Method, Path)] = ...
}

It takes as input the inbound request itself and outputs two chunks split from the request; the method and the path. Through the same syntax as :: that we saw earlier; Scala can match the request as a method on the left and a path on the right. Scala also allows us to match against specific values, so we can further refine our match with an exact method. The path however gets fed to another Path matcher, which decomposes the value further.

1
2
3


object / {
 def unapply(path: Path): Option[(Path, String)] = ...
}

This path matcher consumes a Path and splits to a String component on its right-hand side. The left-hand side returns another Path so we can decompose the path string as much as we need to.

You should see a pattern emerging, and we can start implementing our own. Let’s revisit our desired syntax.

1
2
3


HttpRoutes.of[IO] {
 case Service.Distributor using GET -> Root / "resource" / IntVar(id) =>
}

We can start building our matcher by looking at our preferred inputs and outputs. In this case we will need:

1
2


Input: Request[IO]
Output: (Service , Request[IO])

It takes a request and returns it plus a newly created Service object. Returning the request enables us to continue matching on it down the line. A trivial implementation looks a little like this;

Notice the unapply method actually has a generic type parameter. Pattern matches do not support type parameters so if you include them they must be inferred from the matcher input.

The logic here could become more complex if we wanted it to, for example we could extract a specific user-agent version too. Like AuthedRoutes the best syntax additions are obvious and limited. By changing the order of the outputs we could move our syntax the end of the match rather than the front. Experimenting with the values you extract in your match brings up some interesting possibilities. Take Http4s’ ->> matcher as an example.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13


 /** Extractor to match an http resource and then enumerate all supported methods:
 * {{{
 * (request.method, Path(request.path)) match {
 * case withMethod ->> Root / "test.json" => withMethod {
 * case Method.GET => ...
 * case Method.POST => ...
 * }}}
 *
 * Returns an error response if the method is not matched, in accordance with [[https://datatracker.ietf.org/doc/html/rfc7231#section-4.1 RFC7231]]
 */
 def unapply[F[_]: Applicative](
 req: Request[F]
 ): Some[(PartialFunction[Method, F[Response[F]]] => F[Response[F]], Path)] = ...

Returning a partial-function yields control back to the caller, who can add their own custom behaviour. This allows for some powerful DSLs that can make defining request-dependent service behaviour a breeze.

The big question is… should you? This technique can help simplify a lot of code, but it can make it worse too. Common drawbacks and criticisms of custom DSLs include;

Confusing or pointlessly obtuse syntax (for example, |@| from cats).
They can hide too much. Subtle behaviours introduced through syntax can feel “magic”.
Newcomers need to learn your syntax on top of a language they might not know already.

You and your team decide how far you want to take it. A part of the art of software development includes deciding when to use these kinds of techniques.

“Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.” ― Antoine de Saint-Exupéry

Even small uses can benefit from custom matchers, for example this…

1

 case Left(Some(event)) if event.timestamp > cutOffTime =>

…into this…

1

 case LiveEvent(event) =>

Which reduces cramped matchers into something simple and self-documenting.