This is not a question, more like a blog post. I don't have a blog so I do it here. Feel free to comment or ask questions.

In my past I have learned the programming language Go, and I really started to like the interface type that Go offers, and I wondered, weather it is possible to port this go languge feature to Nim with the aid of macros. For those of you, who know interfaces only from Java, in go interfaces work a bit different and here is the official documentation for it: https://gobyexample.com/interfaces .

First of all, I would like to show you a bit of the advangae of this Go interface type, by comparing it the the options c++ can give you. The c++ version would be almost identical to a Java solution.

virtual methods

struct MyInterface {
    virtual int foo() = 0;
    virtual int bar() = 0;
    virtual ~MyInterface(){}
};

struct MyImplementationA: public MyInterface {
    int foo() override { return 17; }
    int bar() override { return 4; }
};

struct MyImplementationB: public MyInterface {
    int m_foo;
    int m_bar;
    int foo() override { return m_foo; }
    int bar() override { return m_bar; }
};

// polymorphic usage
int foobar(MyInterface* mi) {
    return mi->foo() + mi->bar();
}

#include <iostream>
using std::cout;
using std::endl;

int main() {
    MyImplementationA a;
    MyImplementationB b;
    b.m_foo = 32;
    b.m_bar = 8;
    
    cout << foobar(&a) << endl;
    cout << foobar(&b) << endl;
}

In this version the virtual method dispatch in foobar is done with the vtable. With the declaration of the first virtual method in MyInterface c++ adds a hidden field to the struct, the so called vtable pointer. that points to a struct with method pointers to the method implementations. Both MyImplementationA and MyImplementationB change the value of this pointer to the vtable of teir type. The method invocation foo in foobar opens the vtable, and reads from there which method to invoke. Dependent on the type it is either MyImplementationA::foo or MyImplementationB::foo. Because of this explicit inheritance the interface implementation needs to be newer than the interface itself. Also small objects that need to be compact in memory should better not have virtual methods because of this added hidden field. C# and Java work almost identical, just that in java everything is virtual.

templates

struct MyImplementationA {
    int foo() { return 17; }
    int bar() { return 4; }
};

struct MyImplementationB {
    int m_foo;
    int m_bar;
    int foo() { return m_foo; }
    int bar() { return m_bar; }
};

// polymorphic usage
template <typename T>
int foobar(T* mi) {
    return mi->foo() + mi->bar();
}

#include <iostream>
using std::cout;
using std::endl;

int main() {
    MyImplementationA a;
    MyImplementationB b;
    b.m_foo = 32;
    b.m_bar = 8;
    
    cout << foobar(&a) << endl;
    cout << foobar(&b) << endl;
}

This version beats all problems of the previous version, but not without introducing new problems. There is no hidden field in neither MyImplementationA nor MyImplementationB. foobar is compiled for each type it is called separately. Meaning that the compiler can also inline the the content from foo and bar. Templates in c++ have a huge problem of being recompiled not only for each type, but also for each compilation unit. So depending on the complexity of foobar and the amount of compilation units where foobar is used, this version can really explode compilation time in bigger projects. Nim generics are very much like c++ templates, so I guess they can have this flaw as well. I just don't have big enough projects yet to notice any major concerns with Nim's generics, but that doesn't prove that they do not exist.

Go Interface

package main

import "fmt"

type MyImplementationA struct  {}

func (this *MyImplementationA) foo() int {
        return 17;
}

func (this *MyImplementationA) bar() int {
        return 4;
}

type MyImplementationB struct  {
    m_foo,m_bar int
}

type MyInterface interface {
        foo() int
        bar() int
}

func (this *MyImplementationB) foo() int {
        return this.m_foo;
}

func (this *MyImplementationB) bar() int {
        return this.m_bar;
}

// polymorphic usage
func foobar(mi MyInterface) int {
    return mi.foo() + mi.bar();
}

func main() {
        a := MyImplementationA{}
        b := MyImplementationB{32, 8}
        
        fmt.Println(foobar(&a))
        fmt.Println(foobar(&b))
}

The Go version does neither put a hidden field in the struct datatype or mark anything as virtual. In fact virtual is not a keyword in go nor does it have an equivalent. Methods just become virtual, as soon as the type gets casted into an interface type.

This works, because the vtable is not part of the struct. An instance of MyInterface is actually a tuple of two pointers. The first one is the vtable pointer, and the second one is the pointer to the struct that implements the interface. With this design decision the Go compiler can just create a vtable for any type that has the methods required to implement it. And when the method foobar grows over time, and get's used by a lot of different types, it still get's compiled only once. If you remember one of Go's main features is it's compilation speed.

Now I want the same concept of the Go interfaces in Nim. The idea is to do the same thing in two steps:

  1. build an example case with everything, including vtable and iterface written manually.
  2. write macros that create the exact same vtable and interface that were manually created in step 1

To not make this post too long, I have joined thise two steps into one step. A macro call is in this code followed by a multiline comment that exactly represents the generated source code

import interfacemacros

type
  MyImplementationA = object
    data : array[0,byte]
  
  MyImplementationB = object
    foo : int
    bar : int

createInterface(MyInterface):
  proc foo(this : MyInterface) : int
  proc bar(this : MyInterface) : int

#[
type
  MyInterfaceVtable = object
    foo: proc (this: pointer): int
    bar: proc (this: pointer): int
  
  MyInterface = object
    objet: pointer
    vtable: ptr MyInterfaceVtable

proc foo(this: MyInterface): int =
  this.vtable.foo(this.objet)

proc bar(this: MyInterface): int =
  this.vtable.bar(this.objet)
]#

proc foo(this : ptr MyImplementationA) : int = 17

proc bar(this : ptr MyImplementationA) : int = 4

implementInterface(MyInterface, MyImplementationB)

#[
let MyImplementationBVtable = MyInterfaceVtable(
  foo: proc (this: pointer): int = cast[ptr MyImplementationB](this).foo,
  bar: proc (this: pointer): int = cast[ptr MyImplementationB](this).bar
)

proc MyInterface_InterfaceCast(this: ptr MyImplementationB): MyInterface =
  MyInterface(objet: this, vtable: MyImplementationBVtable.unsafeAddr)
]#

implementInterface(MyInterface, MyImplementationA)

#[
let MyImplementationAVtable = MyInterfaceVtable(
  foo: proc (this: pointer): int = cast[ptr MyImplementationA](this).foo,
  bar: proc (this: pointer): int = cast[ptr MyImplementationA](this).bar
)

proc MyInterface_InterfaceCast(this: ptr MyImplementationA): MyInterface =
  MyInterface(objet: this, vtable: MyImplementationAVtable.unsafeAddr)
]#

proc foobar(mi: MyInterface) : int =
  return mi.foo + mi.bar

var
  a : MyImplementationA
  b : MyImplementationB = MyImplementationB(foo : 32, bar : 8)

echo foobar(MyInterface_InterfaceCast(a.addr))
echo foobar(MyInterface_InterfaceCast(b.addr))

The differences two the Go version are:

  • The interface declaration is in a macro call.
  • The vtables for the different types are requested explicitly with implementInterface(MyInterface, MyImplementation)
  • The conversion between pointer type and interface type is explicit with MyInterface_InterfaceCast
  • error messages when the interface can not be implemented are not really nice

One thing to note is, I could change MyInterface_InterfaceCast from a proc to a converter, and maybe I should to make this even more go like. I just don't like implicit conversions, I think they are scary, because they hide procedure calls from the code reader, and could therefore decrease code maintainability.

And for completion, here are the macros that do the magic:

import macros

macro createInterface*(name : untyped, methods : untyped) : untyped =
  name.expectKind nnkIdent
  
  let
    vtableRecordList = nnkRecList.newTree
    vtableIdent = newIdentNode($name.ident & "Vtable")
    vtableTypeDef = nnkTypeSection.newTree(
      nnkTypeDef.newTree(
        vtableIdent,
        newEmptyNode(),
        nnkObjectTy.newTree(
          newEmptyNode(),
          newEmptyNode(),
          vtableRecordList
        )
      )
    )
  
  var newMethods = newSeq[NimNode]()
  
  for meth in methods:
    meth.expectKind(nnkProcDef)
    let
      methodIdent = meth[0]
      params = meth[3]
      thisParam = params[1]
      thisIdent = thisParam[0]
      thisType  = thisParam[1]
    
    if thisType != name:
      error thisType.repr & " != " & name.repr
    
    let vtableEntryParams = params.copy
    vtableEntryParams[1][1] = newIdentNode("pointer")
    
    vtableRecordList.add(
      nnkIdentDefs.newTree(
        methodIdent,
        nnkProcTy.newTree(
          vtableEntryParams,
          newEmptyNode(),
        ),
        newEmptyNode()
      )
    )
    
    let call = nnkCall.newTree(
      nnkDotExpr.newTree( nnkDotExpr.newTree(thisIdent, newIdentNode("vtable")), methodIdent  ),
      nnkDotExpr.newTree( thisIdent, newIdentNode("objet") ),
    )
    
    for i in 2 ..< len(params):
      let param = params[i]
      param.expectKind(nnkIdentDefs)
      for j in 0 .. len(param) - 3:
        call.add param[j]
    
    meth[6] = nnkStmtList.newTree(call)
    
    newMethods.add(meth)
  
  result = newStmtList()
  result.add(vtableTypeDef)
  result.add quote do:
    type `name` = object
      objet : pointer
      vtable: ptr `vtableIdent`
  
  for meth in newMethods:
    result.add meth
  
  #echo result.repr

macro implementInterface*(interfaceName, implementationCandidateName: typed) : untyped =
  let
    vtableSymbol = interfaceName.symbol.getImpl[2][2][1][1][0]
    vtableRecordList = vtableSymbol.symbol.getImpl[2][2]
  
  let vtableValueSymbol = newIdentNode($implementationCandidateName.symbol & "Vtable")
  
  let
    objectConstructor = nnkObjConstr.newTree(vtableSymbol)
    
    vtableValueDecalaration =
      nnkLetSection.newTree(
        nnkIdentDefs.newTree(
          vtableValueSymbol,
          newEmptyNode(),
          objectConstructor
        )
      )
  
  for identDefs in vtableRecordList:
    let
      methodName = identDefs[0]
      params = identDefs[1][0]
      lambdaBody = quote do:
        cast[ptr `implementationCandidateName`](this).`methodName`()
      call = lambdaBody[0]
    
    for i in 2 ..< len(params):
      let param = params[i]
      param.expectKind(nnkIdentDefs)
      for j in 0 .. len(param) - 3:
        call.add param[j]
    
    # leave out () when not needed
    if call.len == 1:
      lambdaBody[0] = call[0]
    
    methodName.expectKind nnkIdent
    
    objectConstructor.add nnkExprColonExpr.newTree(
      methodName,
      nnkLambda.newTree(
        newEmptyNode(),newEmptyNode(),newEmptyNode(),
        params.copy,
        newEmptyNode(),newEmptyNode(),
        lambdaBody
      )
    )
  
  result = newStmtList()
  result.add vtableValueDecalaration
  
  let castIdent = newIdentNode($interfaceName.symbol & "_InterfaceCast")
  
  result.add quote do:
    proc `castIdent`(this: ptr `implementationCandidateName`) : `interfaceName` = `interfaceName`(
      objet : this,
      vtable : `vtableValueSymbol`.unsafeAddr
    )
  
  # echo result.repr

conclusion: Nim turns out to be a very powerful programming language.

This version here not only gives you a version you can clone right away from github, but also introduces some improvements that will not be inserted into this version anymore.

2016-07-29 15:55:45

Nice work, but please refrain from facts/myths that don't have anything to do with your otherwise very nice article.

This is so bad that several bigger c++ projects completely abandoned templates.

Which ones exactly? LLVM, wxWidgets, Urho3D, Ogre, Unreal Engine 4 and even GCC afaik all use templates. We're not in the 80ies anymore where templates where this special bug ridden feature with stupid syntax that nobody understood.

If you remember one of Go's main features is it's compilation speed.

Go's compile times are nothing special at all. Nim sometimes beats Go's compile times, Delphi always beats it, D can beat it and yet D too has extensive templating everywhere...

2016-07-29 16:17:05

Which ones exactly? LLVM, wxWidgets, Urho3D, Ogre, Unreal Engine 4 and even GCC afaik all use templates. We're not in the 80ies anymore where templates where this special bug ridden feature with stupid syntax that nobody understood.

Ok, I will search for examples, and update the post accordingly. I definitively heared it in talks about c++. And listing big projects that use templates does not beat the argument. Additionally I've heard from Unreal Engine 4 developers, that changing the source code has very long builds, even long incremental builds maybe because of templates, but that not a prove and not my own experience.

Go's compile times are nothing special at all. Nim sometimes beats Go's compile times, Delphi always beats it, D can beat it and yet D too has extensive templating everywhere...

Ok I did not do the benchmark, but correct would be that Go advertises with fast compilation speed. But you are right, in my experience it was not too fast. In my code is was always slower than advertised :/. But I can't exclude that I did something wrong that cause this increased compilation speed, because I never programmed in pure Go, I always had wrappers.

2016-07-29 16:53:25
bpr
Rust, by design, also has C++-like monomorphizing templates, so I think it isn't fair to imply that templates (i.e., generics implemented by monomorphization) are a "considered harmful" language feature these days. As Araq points out, D, especially D2, is a template heavy language which mostly compiles pretty quickly.
2016-07-29 17:24:44

@bpr I never said that monomorphization is a harmful language feature. I just say that someone needs to be aware of it, and that in some contexts this monoporphization is in fact a bad thing.

But in c++ it actually is harmful, because the way c++ deals with templates is, to create all required instances per compilation unit, and then at link time the linker throws all duplicates away. And that I consider as harmful, because the same function might get compiled 100 times and then gets thrown away 99 times. But the problem here is more on the way how c++ compiles its sourcecode, and not on how templates generally work. c++ has the same problem with datatypes that are declared in headers.

2016-07-30 05:19:15

surprisingly, this Go like interface is very similar with what I have implemented when creating high level wrapper for Chromium Embedded Framework in nimCEF.

in my experience, Nim macro not only capable simulate Go like interface, but also interface + simulated inheritance, something not available in Go. Indeed Nim's macro can make Nim a very powerful language.

2016-07-30 15:44:46
@jangko what do you mean with simulated inheritance? In Go you can have an embedded type in a struct. That means your struct has all methods, that the embedded type also has. This is effectively pretty much like multiple inheritance. So what simulated inheritance is not possible in Go?
2016-07-30 18:32:41
+1 Awesome! 2016-07-31 02:13:24

How about:

type
  MyImplementationA = object
  
  MyImplementationB = object
    m_foo,m_bar: int

proc foo(this: MyImplementationA): int =
  return 17

proc bar(this: MyImplementationA): int =
  return 4

proc foo(this: MyImplementationB): int =
  return this.m_foo

proc bar(this: MyImplementationB): int =
  return this.m_bar

proc foobar[T](mi: T): int =
  return mi.foo() + mi.bar()

var a = MyImplementationA()
var b = MyImplementationB(m_foo: 32, m_bar: 8)

echo foobar(a)
echo foobar(b)

2016-07-31 04:45:22
@_tulayang It would be nice to know, where you lost my explanation, or where you started to skip reading it, because the point was to create a single type that can be used to implement foobar, and both MyImplementationA and MyImplementationB can be casted to this single type. Of course using generics for foobar works, too, but that has the disadvantage, that for each type T foobar is called, a new version of that get's compiled. That's not generally a bad thing, but the disadvantage is that in some cases generics, or templates (how generics are called in c++) can cause code bloat.
2016-07-31 13:08:42
<<<••123456••>>>