What is `transparent` in Scala?

Latest Update (Nov 8, 2018): this capability may or may not make it into Scala 3. For several months it looked like it had been pulled, but there are ways in which it might be very helpful, so a tweaked version is back on the table for consideration. When things settle down a bit I will rewrite this article, but for now the key things to keep in mind (based on current PRs) are:

• The transparent keyword is likely to be inline instead.
• When defining a function that refines its return type, you specify the return type with <: instead of the traditional :, to signify that it is actually returning a subtype.
• The guts of the refining function are often written with a new inline match functionality.

See this PR for a sense of the current state of play, and this file for a good example, similar to the code I describe below.

Original Article:

Just got back from ScalaDays, which was great fun as always. But the conference started with me getting thrown for a mild loop.

Martin’s introductory keynote was about Scala 3, and what we could expect from it. He talked about all the lovely stuff that was coming in — enums and opaques and more consistency to make the language easier to reason about and use — and I nodded along contentedly, having read into all of it over the past year.

And then he introduced the keyword transparent, and did a demo, which kind of blew my mind. This was the first I’d heard of it, so I spent the past several days talking to folks and investigating code, trying to understand what it does.

This is a writeup of my current understanding of transparent, based on what I’ve found. It comes with a bunch of caveats:

• I am not a compiler expert of any sort, so any or all of the below may be mistaken. Corrections by those who know what they’re talking about are welcome.
• This is all extremely new and experimental. It not only isn’t part of a Dotty release yet, I don’t believe it’s even merged into the master branch. It mostly appears to live in this open pull request, which is rather fun to poke around in.
• This writeup is based on me reading through that pull request (particularly the tests) and talking to folks, but it’s a bit third-hand.

Really, I wouldn’t even bother writing about this, except that nobody else has apparently done so yet, and it seems worth putting out there. And since Martin brought it up so publicly at ScalaDays, I figure it’s fair game to talk about it.

The pain point (as I understand it)

One of the most powerful, and increasingly essential, libraries in the Scala ecosystem is Shapeless. Shapeless introduced a lot of crucial abstractions such as HLists and Generics that allow higher-level libraries like Circe to handle complex problems like serialization with remarkably little boilerplate. It permits type-level programming — reasoning about code at the compiler level, and leveraging the conclusions to remove boilerplate while remaining fully type-safe.

The only problem with Shapeless is that, frankly, it’s fairly hard to understand. It performs its magic by leveraging the power of Scala’s type system, which is essentially a logic-programming language. While The Type Astronaut’s Guide has done much to teach Shapeless to far more people, it’s still not an easy or natural way of thinking about things for most of us, and the syntax is pretty convoluted.

So transparent is one of several enhancements to the language, which make this sort of power programming more idiomatic, allowing you to do some of this type manipulation using functional-programming techniques instead of logic-programming ones.

The term `transparent`

Far as I can tell, the name for the new keyword was chosen in deliberate contrast to the more-mature new feature opaque.

It makes a certain sort of sense. The point of opaque is that it lets you define a new type where the details are intentionally hidden. For those not familiar with it, opaque is going to largely replace the way we’ve been using AnyVal — it lets you create a new type that is actually an Int or String or whatever under the hood, with guaranteed non-boxing, but totally hides that underlying type except under certain circumstances.

By contrast, transparent is a new keyword that you can apply to def and val, which lets the compiler infer and expose more type information than what is written into the signatures.

What?

As I understand it, transparent is basically inline on steroids. It inlines a function, but beyond that it encourages the compiler to figure out the resulting types more precisely than what you originally declared, and lets calling code use those deductions.

Right now, the best way to understand this seems to be looking at the tests in the new PR — in particular, I recommend looking at the file typelevel.scala. I’m going to quote some of the interesting bits from that, and describe what I think is happening.

Here’s a simple but crucial type:

trait Nat {  
  def toInt: Int
}

This represents a natural number at the type level, so we can talk about things like indexes in types. This is the type-level concept of “zero”:

case object Z extends Nat {
  transparent def toInt = 0
}

Now we encounter transparent for the first time. If we ever call Z.toInt, it will be inlined, but more than that, it will be refined: the type when you call Z.toInt isn’t Int, it’s the more-precise type 0. (One of the nice enhancements coming soon is literal types, so you can treat Int and String literals as singleton types.)

Okay, we have zero — how do we get the rest? We treat them as successors to zero:

case class S[N <: Nat](n: N) extends Nat {
  transparent def toInt = n.toInt + 1
}

So the way we express 2 in this system is as S(S(Z)) — that is, 0 + 1 + 1. And again, toInt is declared as transparent, which means that you can say:

val myTwo: 2 = S(S(Z)).toInt

and the compiler would agree with you: it inlines those case class calls and the recursive toInt calls, so it knows at the type level that it is doing 0 + 1 + 1. The result is the type 2. Neat!

HList

Now let’s take a look at the type that is the heart of most type-level programming: HList. If you haven’t already seen it, this is a sort of hybrid of Lists and Tuples.

Like a Tuple, it is heterogeneous. In a Tuple, each element can be of a different type, and the Tuple depends on each of these types being used in the correct position. For instance, when you say:

val myTuple = (120, "hello")

The type of myTuple is (Int, String). It’s still strongly-typed — the first element must be Int, the second must be String. But Tuples can’t vary in length: you can’t concatenate a Float to the end of that and get a (Int, String, Float).

On the other hand, you have Lists. These can vary in length, but have to be more or less homogeneous. So you can concatenate two Lists:

val twoLists = List(1, 2, 3) ++ List(4, 5, 6)
assert(twoLists == List(1, 2, 3, 4, 5, 6))

But you can’t mix multiple different types in the same List. (Yes, yes, there is also variance, but that doesn’t really affect the point.)

HLists combine both of these qualities. Like List, you can add and remove members; like Tuple, the member at each position must be of a specific type. So you can say, for example:

val firstH = 120 :: "hello" :: HNil
val secondH = 2.3 :: "there" :: HNil
val together: Int :: String :: Float :: String :: HNil 
  = concat(firstH, secondH)

This ability to flexibly manipulate strongly-typed HLists opens a world of power. See the Shapeless documentation for examples of things you can do with this sort of programming.

HLists with `transparent`

Conveniently, the tests for transparent provide an example implementation of HList that we can look at. First, there is the base type:

trait HList {
  def length: Int
  def head: Any
  def tail: HList
  transparent def isEmpty: Boolean =
    length == 0
}

Note that isEmpty is transparent. So its type is Boolean, but that might get refined further. We can see that in action when we look at HNil, the empty HList:

case object HNil extends HList {
  transparent def length = 0
  def head: Nothing = ???
  def tail: Nothing = ???
}

The head and tail aren’t implemented, because this is the empty list: they’re meaningless. But length is now transparent. So let’s work out what that means for isEmpty, which we saw above. When we call HNil.isEmpty, the compiler can inline it as:

HNil.isEmpty        ==> 
  HNil.length == 0  ==>
  0 == 0            ==>
  true

So at compile time, the compiler can prove that that call is true. Moreover, it can return it as the type true, without that ever being hard-coded. And while I don’t know the compiler internals, I would bet that that is hellaciously useful for optimizations.

How about when the list isn’t empty? The HCons type represents a node in the list, which has a value hd of type H:

case class HCons [H, T <: HList](hd: H, tl: T) extends HList {
  transparent def length = 1 + tl.length
  def head: H = this.hd
  def tail: T = this.tl
}

length is transparent again, so let’s play with that. Say we have an HList like this:

val xs = HCons(1, HCons("a", HNil))

We can now write:

transparent val len1 = xs.length
val len1a: 2 = len1

Let’s work through that:

transparent val len1 = xs.length        ==>
  = HCons(1, HCons("a", HNil)).length   ==>
  = 1 + HCons("a", HNil).length         ==>
  = 1 + 1 + HNil.length                 ==>
  = 1 + 1 + 0                           ==>
  = 2

And since len1 is declared as transparent, that means that usages of it get inlined, and it not only returns the value 2, but the type 2. So that last line compiles down to:

val len1a: 2 = 2

Again, we get to see the heart of transparent — it works through the expression (either a def or a val), works through the full inlining, infers all the types and then leaks the resulting types out from the expression.

The result is that, at the call site, we can wind up seeing a more specific type than what is actually declared in the signature. Nat.toInt doesn’t actually return Int, it returns the specific subtype of Int that represents the literal number we want. (Provided, of course, that enough information is available at the call site.) Same for HList.length. And whileHList.isEmpty is declared as Boolean, it actually returns the literal subtype true or false of whether it is empty, at compile time.

Conclusions

I’ll repeat again: this is all guesswork, based on what I’ve heard and what I’m reading. I recommend reading through typelevel.scala yourself (there’s a good deal more there), and I’d love it if somebody who knows this stuff better can say whether I’m on the right track.

But assuming I’m correct, this is one of the most exciting innovations in Scala 3 — sort of like a supercharged version of type inference. Combined with the other features of Scala 3, it looks likely to make type-level programming much easier and more intuitive, especially if they can come up with a similarly sensible approach to Generics. It’s mostly going to be used by library authors, but I suspect it’ll be great for that purpose. Fingers crossed that this feature works out in the long run…