Tomas Basham’s Blog

Diagnosing Pods with Kubernetes

2018-01-22T00:00:00+00:00

Labels are the mechanism in Kubernetes through which system objects may be given an organisation structure. A label is a key-value pair with well defined restrictions concerning length and value used as an identifier and mapped onto system objects making them more meaningful within the system as a whole.

Whilst labels (and selectors) underpin how Pods are managed within a Deployment they can be used in other ways more conducive to the diagnosis of the containers that make up Pods.

When there is a problematic Pod running within a cluster it may not be desirable to destroy it without first understanding what went wrong. Instead the Pod should be removed from the load balancer and inspected whilst no longer serving traffic.

This can be accomplished through the use of labels and selectors within a Kubernetes manifest files describing a Deployment. In particular a label such as serving: true may indicate to Kubernetes that a Pod should be placed within the load balancer and should be serving traffic.

Note: The -L option in the following command specifies the serving label should be included in the table output by kubectl.

$ kubectl -n myapp get pods -L serving
NAME                     READY     STATUS    RESTARTS   AGE       SERVING
api-441436789-7qzv0      2/2       Running   0          3d        true
api-441436789-dwg2q      2/2       Running   0          3d        true
client-760894609-ggw1b   1/1       Running   0          3d        true
client-760894609-s8blr   1/1       Running   0          3d        true

$ kubectl -n myapp describe service/api
Name:                     api
Namespace:                myapp
Labels:                   <none>
Annotations:              cloud.google.com/load-balancer-type=internal
                          kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"cloud.google.com/load-balancer-type":"internal"},"name":"api","namespace":"myapp"},"spec..."
Selector:                 app=myapp,serving=true,tier=api
Type:                     LoadBalancer
IP:                       10.7.241.228
LoadBalancer Ingress:     10.132.0.6
Port:                     http2  80/TCP
TargetPort:               %!d(string=esp-port)/TCP
NodePort:                 http2  31364/TCP
Endpoints:                10.4.2.19:9000,10.4.2.21:9000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

For instance lets say the Pod named api-441436789-7qzv0 is returning an increased number of 404 responses which has been identified as abnormal behaviour. This Pod is a prime candidate for inspection, but doing so whilst it is still serving traffic (and thus many 404 errors) is undesirable. Replacing the Pod with a fresh one may solve the problem temporarily, but finding the root cause of the issue should be the goal. Therefore to remove this Pod from the load balancer it’s labels must be edited in place.

$ kubectl -n myapp label pods/api-441436789-7qzv0 --overwrite serving=false

The Replication Controller backing the Deployment will spin up a new Pod to replace the one taken from the load balancer whilst the problematic Pod will remain active for inspection but will not be available to serve traffic.

$ kubectl -n myapp get pods -L serving
NAME                     READY     STATUS    RESTARTS   AGE       SERVING
api-441436789-7qzv0      2/2       Running   0          3d        false
api-441436789-dwg2q      2/2       Running   0          3d        true
api-441436789-lh8ht      2/2       Running   0          8s        true
client-760894609-ggw1b   1/1       Running   0          3d        true
client-760894609-s8blr   1/1       Running   0          3d        true

$ kubectl -n myapp describe service/api
Name:                     api
Namespace:                myapp
Labels:                   <none>
Annotations:              cloud.google.com/load-balancer-type=internal
                          kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v3","kind":"Service","metadata":{"annotations":{"cloud.google.com/load-balancer-type":"internal"},"name":"api","namespace":"myapp"},"spec..."
Selector:                 app=myapp,serving=true,tier=api
Type:                     LoadBalancer
IP:                       10.7.241.228
LoadBalancer Ingress:     10.132.0.6
Port:                     http2  80/TCP
TargetPort:               %!d(string=esp-port)/TCP
NodePort:                 http2  31364/TCP
Endpoints:                10.4.2.21:9000,10.4.2.24:9000
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

Note: In the above snippet the endpoints that back the service have changed to reflect the replacement of the problematic Pod.

Now an interactive terminal session can be made with a container running in the Pod without affecting traffic.

kubectl -n myapp exec -it api-441436789-7qzv0 -c api /bin/bash
root@api-441436789-7qzv0:/usr/src/app#

When the problem has been diagnosed and possibly fixed the Pod can be returned to the load balancer or more likely just destroyed.

Protocol Buffers and Optional Fields

2017-09-13T00:00:00+00:00

Protocol Buffers are a serialisation solution developed by Google providing a platform and language neutral mechanism to send and receive structured data across the wire. Protocol Buffers encode data into dense binary objects that reduce packet sizes, enabling faster data exchange but with the disadvantage that the binary objects are not easily human readable.

Protocol Buffers allow developers to define how they want their application data to be structured and the Remote Procedural Calls (RPC) to be made available for a particular service. An example of a Protocol Buffer (specifically proto3) definition follows with a “Greeter” service that takes a user’s name and returns a customised greeting.

syntax = "proto3";

package greeter;

service Greeter {
  rpc Greet(GreetRequest) returns (GreetResponse);
}

message GreetRequest {
  string name = 1;
}

message GreetResponse {
  string message = 1;
}

Very often we rely on inconsistent code at the boundaries of our applications where rarely we enforce strict typing and structure of the data we exchange. Being able to encapsulate the business semantics of an application within a simple definition strengthens those boundaries providing a means to enforce business logic. This functions well when all data is required to be present in each message, however in less trivial applications business logic demands for the ability to define values that may be nullable; something not possible with proto3 using primitive data types. In contrast previous versions of Protocol Buffers, namely proto2, allowed for optional fields providing nullable support for any field type.

To understand why, it may help to think about for what reasons proto2 and proto3 both omit fields from the encoded data. In proto3 fields always have a value and as such are never left unset. Because of this proto3 can achieve a smaller payload by not transferring fields set to their default values. This incurs a form of compression, saving a few bytes on message exchange.

The proto2 specification, in addition to the above, keeps track of whether the current value was explicitly set on a field. It is this feature of proto2 that allows for the optional field types. If the current value was not explicitly set then the value is not transferred. This of course carries a small penalty of needing extra storage space to transfer flags indicating fields that have been explicitly set.

In light of this proto3 has been designed to distinguish between the absence of a primitive typed field and its default value through the use of the official value wrapper types that form part of the proto3 “standard library”.

syntax = "proto3";

package greeter;

import "google/protobuf/wrappers.proto";

service Greeter {
  rpc Greet(GreetRequest) returns (GreetResponse);
}

message GreetRequest {
  google.protobuf.StringValue name = 1;
}

message GreetResponse {
  string message = 1;
}

The main difference here is the addition of the StringValue submessage that is a simple wrapper around a string field.

message StringValue {
  string value = 1;
}

In proto3, submessages can be set to nil allowing developers to use wrappers for fields where the value may be optional. This adds a small amount of overhead since wrapper values end up consuming an extra byte per field because of the additional message layer. In addition the presence of the field must be checked before it is used.

Despite this, one still cannot guarantee that the value in the wrapper was explicitly set or defaulted to its zero value. However this moves the problem of supporting optional values further away from the proto3 specification and more toward developer conventions. Typically if a submessage is nil then an application consuming the message can assume the field was deliberately (or perhaps accidentally) omitted. If however the submessage has a non-nil value then it must be assumed that whatever value is held by the wrapper was intended to be consumed.

Kubernetes Networking

2017-06-30T00:00:00+00:00

In the software development scene today, containerisation - and Docker in particular - is hard to ignore. However when an application needs to scale horizontally it falls upon container orchestration systems to handle the complex interconnections between distributed nodes and the containers that they manage.

It is without doubt that Kubernetes has prevailed as the dominant container orchestration system, but the networking model that it implements is often difficult to understand. Before discussing how Kubernetes approaches networking it is worth understanding how the Docker networking model is implemented to acknowledge some of the issues.

The Docker Model

The Docker networking model is somewhat flexible, offering support for multiple solutions through the use of network drivers. However in the default case Docker imposes a bridge networking model using a host-private networking scheme. Docker creates a virtual bridge, called docker0, in the root network namespace and allocates a subnet from one of the private address blocks defined in RFC1918 (typically 172.16.0.0/12) to that bridge. For each container managed by Docker a virtual network namespace is created to isolate the container from the host networking device. Each namespace is attached to the bridge through a virtual Ethernet device (veth) and mapped to appear as eth0 from within the container. The virtual Ethernet device is then allocated an IP address from the address range assigned to the bridge.

The result of this setup is that each container Docker manages can communicate with all other containers providing they are connected to the same virtual bridge. By extension since the bridge is configured behind the node’s own Ethernet device the containers must also be on the same machine. In addition because Docker is configured to use the same IP address range on every node it follows that containers across nodes may be assigned the same IP address. This makes containers unable to communicate with each other across nodes out-of-the-box.

One way for containers across nodes to communicate it to carefully coordinate the allocation of ports on the node’s own IP address and then forward packets to the respective container. This can be rather errors prone and often lead to high contention.

Below is a diagram depicting how Docker manages virtual network namespaces for each container. From this it can be seen how packets would have to traverse through the virtual bridge to provide connectivity between containers on the same node.

The Kubernetes (IP per Pod) Model

The Kubernetes networking model does not differ much from the Docker model seen above. However Kubernetes demands a flattening of the IP address space, dictating that all containers (and their respective nodes) should be able to communicate with each other without the use of Network Address Translation (NAT). How this is achieved is of no concern to Kubernetes and may likely be implemented differently across infrastructure providers. For instance simple L2 ARP lookups across a switching fabric could achieve this, or alternatively L3 IP routing, or an overlay. Providing this demand is respected Kubernetes should be able to run across a network.

Unlike the Docker model, Kubernetes assigns IP addresses at the Pod level, where by default a Pod is allocated a private IP address within the network namespace address range.

Pods themselves provide an isolated, shared network namespace for their content (containers) meaning containers within a Pod can communicate by making a request to localhost. This consequently means that each container within a Pod must coordinate port usage. However it is typically considered best practice to isolate a single container within a single Pod so this issue tends to be moot. On the occasion there exists one or more containers with a hard dependency on each other they can be placed within a single Pod but should not likely need to contest for unallocated ports.

To demonstrate this I have created a single Pod running 2 containers. The kubectl command prints out the Pods details.

Note The -l options in the following command specifies that only Pods with a tier label set to api should be selected for the output.

$ kubectl -n myapp get pods -l tier=api
NAME                   READY     STATUS    RESTARTS   AGE
api-4146221093-7pv1p   2/2       Running   0          1d

$ kubectl -n myapp describe pod/api-4146221093-7pv1p
Name:           api-1
Namespace:      myapp
Node:           gke-development-cluster-default-pool-9bf27fcd-vqvz/10.132.0.2
Start Time:     Thu, 28 Jun 2017 08:59:10 +0100
Labels:         app=myapp
                pod-template-hash=4146221093
                serving=true
                tier=api
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"myapp","name":"api-4146221093","uid":"f305e98b-f19f-11e7-a706-42010a840036","apiV..."
Status:         Running
IP:             10.4.2.19
Created By:     ReplicaSet/api-4146221093
Controlled By:  ReplicaSet/api-4146221093
Containers:
  ...

The above Pod can be accessed by making a request to its IP address (10.4.2.19). Since the cluster is running on Google Kubernetes Engine (and by extension Google Compute Engine) where by default a Pod is given an internal IP address it can only be accessed from a machine within the same Google Cloud Platform project.

$ echo -e "GET / HTTP/1.1\n" | nc 10.4.2.19 9292
HTTP/1.1 200 OK
Content-Type: text/html;charset=utf-8
Content-Length: 0
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Date: Mon, 28 Jun 2017 18:23:32 GMT
Connection: Keep-Alive

However having applications make requests to this IP address would be futile in the long run. Since Pods are by nature ephemeral the IP address assigned to this Pod may not be the same IP address assigned to a Pod in the future. Instead there needs to be a layer in front of the Pods to maintain a stable endpoint.

The Service Abstraction

Services are an abstraction of stability for Pods providing a persistent endpoint representing a set of containers behind it. This is achieved with a Virtual Internet Protocol (VIP) address and enables Kubernetes to deal with changes to the cluster topology dynamically without effecting user perceived uptime. Clients now only need know the VIP in order to talk to the Pods behind the Service. This allows for rolling updates where Pods may be completely replaced, likely allocated different IP addresses, without the client realising what actions Kubernetes has taken.

The default implementation of the Service abstraction is the kube-proxy that runs on each node in a cluster. Its main responsibility is to query the Kubernetes API server and configure iptables to forward packets to the correct destination Pods (backends). It is able to configure iptables to perform simple TCP and UDP stream forwarding or round robin TCP and UDP forwarding across a set of backends. Despite its name kube-proxy is not a proxy - once upon a time it was a proxy, now it is a controller. In fact it does not touch the packets traversing the Kubernetes managed network.

To demonstrate this I have created a Service that creates a persistent endpoint for a single Pod. The kubectl command prints out the Services details.

$ kubectl -n myapp get services
NAME      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
api       ClusterIP   10.7.241.228   <none>        80/TCP   1d

$ kubectl -n myapp describe service/api
Name:              api
Namespace:         myapp
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"api","namespace":"myapp"},"spec":{"ports":[{"name":"http2","port":80,"protocol..."
Selector:          app=myapp,serving=true,tier=api
Type:              ClusterIP
IP:                10.7.241.228
Port:              http2  80/TCP
TargetPort:        %!d(string=esp-port)/TCP
Endpoints:         10.4.2.19:9000
Session Affinity:  None
Events:            <none>

Now the same Pod can be reached by making a request to the VIP (10.7.241.228) assigned to the Service.

$ echo -e "GET / HTTP/1.1\n" | nc 10.7.241.228 80
HTTP/1.1 200 OK
Content-Type: text/html;charset=utf-8
Content-Length: 0
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Date: Mon, 28 Jun 2017 18:23:32 GMT
Connection: Keep-Alive

By logging onto the node managing the Pod (although any of the nodes would produce the same output) we are able to see the list of iptables rules that have been created by kube-proxy.

$ sudo iptables-save | grep api
-A KUBE-SEP-JREY6CMQUZ2KMT2G -s 10.4.2.19/32 -m comment --comment "myapp/api:http2" -j KUBE-MARK-MASQ
-A KUBE-SEP-JREY6CMQUZ2KMT2G -p tcp -m comment --comment "myapp/api:http2" -m tcp -j DNAT --to-destination 10.4.2.19:9000
-A KUBE-SERVICES ! -s 10.4.0.0/14 -d 10.7.241.228/32 -p tcp -m comment --comment "myapp/api:http2 cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.7.241.228/32 -p tcp -m comment --comment "myapp/api:http2 cluster IP" -m tcp --dport 80 -j KUBE-SVC-CMKF3ZHDEOE3K64F
-A KUBE-SVC-CMKF3ZHDEOE3K64F -m comment --comment "myapp/api:http2" -j KUBE-SEP-JREY6CMQUZ2KMT2G

When a Pod wants to request this Service any packets are forwarded from the network bridge, through iptables and onto the destination Pod. Specifically kube-proxy has created rules that perform a Destination Network Address Translation (DNAT) on all packets destined for 10.7.241.228/32 over TCP on port 80 and rewrites the destination address to the Pod running the container. In this case iptables rewrites the destination address to 10.4.2.19 on port 9000.

It is important to note that iptables, in addition to performing a DNAT, also creates a record in the connection tracking table. This allows iptables to track address translations based on a 5-tuple schema later used to reverse the translation when a response is received.

Scaling Up

So far we have only seen what happens with a single Pod. However most distributed applications require redundancy, adding further complexity. To demonstrate this I have increased the number of replicas for the Pod from above to 2. The kubectl command prints out the Services details.

$ kubectl -n myapp describe service/api
Name:              api
Namespace:         myapp
Labels:            <none>
Annotations:       kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"api","namespace":"myapp"},"spec":{"ports":[{"name":"http2","port":80,"protocol..."
Selector:          app=myapp,serving=true,tier=api
Type:              ClusterIP
IP:                10.7.241.228
Port:              http2  80/TCP
TargetPort:        %!d(string=esp-port)/TCP
Endpoints:         10.4.1.14:9000,10.4.2.19:9000
Session Affinity:  None
Events:            <none>

Once the new Pod has been created the Service has 2 backends where each Pod may very well be on different nodes. However because of the rules Kubernetes demands these Pods can both talk to one another. When this new Pod was created kube-proxy created new rules in the iptables across all the nodes in the cluster. These rules forward packets to the new Pod when the Service is requested.

$ sudo iptables-save | grep api
-A KUBE-SEP-A2E6GM52XHOS2X76 -s 10.4.1.14/32 -m comment --comment "myapp/api:http2" -j KUBE-MARK-MASQ
-A KUBE-SEP-A2E6GM52XHOS2X76 -p tcp -m comment --comment "myapp/api:http2" -m tcp -j DNAT --to-destination 10.4.1.14:9000
-A KUBE-SEP-JREY6CMQUZ2KMT2G -s 10.4.2.19/32 -m comment --comment "myapp/api:http2" -j KUBE-MARK-MASQ
-A KUBE-SEP-JREY6CMQUZ2KMT2G -p tcp -m comment --comment "myapp/api:http2" -m tcp -j DNAT --to-destination 10.4.2.19:9000
-A KUBE-SERVICES ! -s 10.4.0.0/14 -d 10.7.241.228/32 -p tcp -m comment --comment "myapp/api:http2 cluster IP" -m tcp --dport 80 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.7.241.228/32 -p tcp -m comment --comment "myapp/api:http2 cluster IP" -m tcp --dport 80 -j KUBE-SVC-CMKF3ZHDEOE3K64F
-A KUBE-SVC-CMKF3ZHDEOE3K64F -m comment --comment "myapp/api:http2" -m statistic --mode random --probability 0.50000000000 -j KUBE-SEP-A2E6GM52XHOS2X76
-A KUBE-SVC-CMKF3ZHDEOE3K64F -m comment --comment "myapp/api:http2" -j KUBE-SEP-JREY6CMQUZ2KMT2G

Now Kubernetes must make a choice as to which backend to forward packets. iptables will pick one of the backends at random, based on some statistic condition. In this case each Pod has a 50% chance of being selected. From here the process is as before; iptables will perform a DNAT, add a record to the connection tracking table, and forward packets to the destination Pod.

The root network namespace now acts as a distributed load balancer. This is true because the same set of rules are configured on each of the nodes in a cluster, so a Service can be “discovered” in the same way from every Pod, regardless of the node that it runs on.

External Resources

The New Full Stack Developer

2017-04-18T00:00:00+00:00

I have recently been on the look out for new employment opportunities, where I have noticed most applications (as opposed to job descriptions a few years ago) require you to be what they regard as a “full-stack” developer. It is assumed that “full-stack” carries a well known definition, accepted by all employers, where one can easily identify whether they fulfil this criteria. However looking closer at current trends in the development and operations space (not DevOps) it follows that perhaps the definition should be expanded.

I’m sometimes baffled by friends and colleges that claim to be “full-stack” developers having as little as a few months real programming experience, or where the foundation of their ability is sourced from online programming courses. It begs the question as to whether they believe software development is 100% programming and that knowing Ruby on Rails and React makes them a “full-stack” developer.

Looking more broadly at the software development scene, containerisation – and Docker in particular – is hard to ignore these days. Coupled with container scheduling solutions such as Kubernetes there is a huge push to hire for positions where the successful applicant can demonstrate skills in these technologies in additional to more traditional software development skills.

I propose a change to the definition of a “full-stack” developer to include the ability to handle operations too. This could mean Puppet, Chef, Capistrano, Kubernetes, networking or all of the above but some experience with servers and deployment should, in my honest opinion, be essential to the definition of “full stack”. Especially now where the line between developer and operations has become a little blurred (infrastructure as code) and where it should be the responsibility of the developer to deploy and maintain their code from conception to production.

I would be interested to read what others think of this and whether my opinion is well grounded.

5 Things I Hate About Ruby

2017-03-06T00:00:00+00:00

I find myself more and more lately banging on about how much better Ruby is when compared to the other languages used by my colleagues (i.e. PHP). I know, I know I am truly a Ruby fanboy. I much favour being able to write my applications in a more “plain English” syntax which offers self documenting and cleaner code.

However I don’t really believe you can trust an advocate who doesn’t know enough to find something wrong with what they are advocating. Given that I use, love and shout Ruby’s praises on almost a daily basis, listing out some of it’s bug bares seems like a good exercise in humility.

5. Private is not private

In Ruby there is no such thing as private or protected scope, at least semantically speaking. The private and protected keywords only act as hints to the interpreter not to allow access to methods outside their intended scope. This leads to a language feature that is not entirely transparent, but equally as trivial.

It would appear that the language instead guides developers down the “right” path by making private method invocations more annoying, throwing exceptions, but does not stop you from doing so. I believe having this relaxed attitude to method visibility opens the doors to poor code design. Personally I would prefer to be forced to code in a certain way than have a mechanism to guide my coding best practices.

To invoke a private method on an object you may simply use the Object.send method.

4. Confusing operators?

Ruby has two different, but confusing, set of operators. Those being the widely used && and || which are present in almost every modern computer language, and the more English style and and or. To be honest this video by Avdi Grimm does a far better job of explaining the differences than I can, but it suffices to say that this syntax is copied from Perl whereby the English style operators have a lesser precedence over the more traditional style operators.

To be fair both set of operators have different use cases so it is hard to hate on this feature of Ruby, however it is a very confusing syntax so anybody would be excused if thinking that they in fact were synonymous.

3. Optional perentheses

Ruby does not require that you place parentheses around method arguments. Although this saves on keystrokes it can be detrimental to readability and in some cases cause your application to not behave as expected. Take the following:

x = 5

puts (0..10).include? x ? 'yes' : 'no'

# is equivalent to

puts (0..10).include?(x ? 'yes' : 'no')

# is equivalent to

puts (0..10).include?('yes')

# is equivalent to

puts false #=> false

Clearly this was not the intended result. The Ruby interpreter has taken x ? 'yes' : 'no' as a single argument to the include? method whereas the intention was only to take x. The correct way to have written this was with parentheses:

x = 5

puts (0..10).include?(x) ? 'yes', 'no'

# is equivalent to

puts true #=> true

The generally accepted rule is to omit parentheses around parameters for methods that are part of an internal DSL (e.g. Rake, Rails, RSpec); methods that are with ‘keyword’ status in Ruby (e.g. attr_reader, puts) and those which take no arguments. Use parentheses for all other method invocations.

2. Conventions

There are certain conventions to the Ruby language, but this point concerns itself around method naming. For example any method ending with a bang (!) indicates that the method will modify the object it’s called on. Similarly any method ending with a question mark indicates that the method will return a boolean - true or false. Although this is widely accurate, there are some instances where this does not hold true. Take for example the Float.infinite? method. Instead of returning true or false it instead returns a trinary result of either nil, 1 or -1.

What is the point of having conventions when they can be ignored at any given time. This now requires developers to remember a series of methods that do not follow conventions, which defeats the point of having the convention in the first place.

1. Defining private class methods

This touches on the first point in this post where private methods are not actually private. However this point has driven me crazy in the past so was worth writing about. There are 2 main ways to define class methods. The first is using self.method_name and the second is as a singleton using class << self. Both achieve the same result, defining a method on, what is known in the Ruby community, as the “Eigenclass”. This is effectively an anonymous class that Ruby creates and inserts into the inheritance hierarchy to hold the class methods.

The problem here is when defining private class methods. You would expect to write something along the lines of this:

class SomeClassWithPrivateMethods
  def self.method_one
    puts 'method one is public'
  end

  private

  def self.method_two
    puts 'method two is public'
  end
end

This however does not make method_two private. The private keyword only affects methods on the class instance, so when we define a class method under the private keyword it does nothing.

Using the singleton approach (class << self) we are defining instance methods on the “Eigenclass”, which are just class methods of the containing class. Confusing, right?

class SomeClassWithPrivateMethods
  class << self
    def method_one
      puts 'method one is public'
    end

    private

    def method_two
      puts 'method two is private'
    end
  end
end

This now works as expected, but adds that extra level of complexity to the Ruby language that does not really offer any benefits as a developer. I would much prefer using self.method_name if only it was affected by the private keyword.

Git Renegade - Delete Remote Tags

2017-01-10T00:00:00+00:00

This is not something considered best practice but if the need arises that a tag must be removed from a remote repository server then here is how to delete it.

If you have a remote tag named branch-name it can be deleted with:

$ git tag -d branch-name
$ git push origin :refs/tags/branch-name

This will delete the tag both locally and remotely.

Swift 3 - Pattern Matching

2016-12-04T00:00:00+00:00

Whilst working on an iOS app, I was able to really jump into some Swift 3 features that were very new to me. Swift 3 has so many advantages over Objective-C but the one I am most impressed with is pattern matching.

Having dabbled a bit with Erlang and other functional programming languages in the past I can immediately see the benefits of this functional paradigm within Swift. It is rather advantageous because Swift now allows us to solve problems perhaps beter suited to functional programming languages (at least where pattern matching is concerned), and we may take examples from these other language to adapt them to work with Swift. So without further ado, lets talk about pattern matching.

The Model

Say you are modelling your favourite burger. Here’s mine. Each topping could be represented as a separate case within an enum.

  enum BurgerTopping {
    case Bun(type: String, seeds: Bool)
    case Patty(type: String)
    case Cheese(type: String)
    case Pickle
    case Tomato
  }

You get the idea. Add whatever toppings take your fancy.

The Humble Switch

In essence we have been using very basic pattern matching for a long time in languages such as Objective-C, C, Java, and many others through the form of switch statements. Here we tend to match strings with other strings or integers with other integers. But now we can take the humble switch statement and put it on steroids.

Using pattern matching switch statements can be used to match more complex structures containing placeholder variables, and bind these variables to real values if they match:

  func add(_ topping: BurgerTopping) {
    switch topping {
    case let .Bun(type, _):
      print("\(type) bun with or without seeds. The mystery is killing me")
    case let .Patty(type):
      print("Juicy. Delicious. Succulant \(type) patty")
    case let .Cheese(type):
      print("You genius! \(type) Cheese. Sounds great!")
    case .Pickle:
      print("Pickles? Ok, if you're sure")
    case .Tomato:
      print("Gotta love those tomatoes")
    default: break
    }
  }

When the BurgerTopping instance matches one of the cases, lets take the second case as an example, then a new variable type is created and the associated value is bound to this variable.

Notice also the use of the wildcard pattern _ in the first case, which indicates the presence of a variable but we don’t really care what it is. In the above code I have used it in place of the seeds value. This is because I don’t intend on using it in the proceeding statement.

Also note the fourth and fifth cases where no variables have been bound. Pickle and Tomato have no associated values so we don’t need to add the let keyword. Furthermore if we were to write case let .Bun(_, _) this would be equivalent to case .Bun as we dont need to bind any values.

Where and Fixed Values

As you can see there are a few way in which pattern matching can be made useful for this use case. The BurgerToppings could be made a lot more complex or you could be modelling something entirely different (i.e. a signup form where each cell in a tableview could be represented).

But we can go further. Where I had included a wildcard in place of the seeds value I now want to have a separate case for when seeds is either true or false. We can do this with the where clause. This is added to the end of a case as shown below:

  func add(_ topping: BurgerTopping) {
    switch topping {
    case let .Bun(type, seeds) where seeds == true:
      print("\(type) bun with seeds. Yeeha!")
    case let .Bun(type, seeds) where seeds == false:
      print("Ooooo. A lovely \(type) bun. Good choice!")
    }
  }

Here the first case will match with a Bun where its seeds value is equal to true, whereas the seconds case will match with a Bun where its seeds value is equal to false. Of course this is a very trivial equality whereby the value can only ever be true or false and can alternatively be written with Fixed Values:

  func add(_ topping: BurgerTopping) {
    switch topping {
    case let .Bun(type, true):
      print("\(type) bun with seeds. Yeeha!")
    case let .Bun(type, false):
      print("Ooooo. A lovely \(type) bun. Good choice!")
    }
  }

This does the exact same thing as the first case except instead of using where I have explicitly attempted to match a constant value.

If and Guard Case

The switch statement is not the only to benefit from pattern matching. It may also be used with if case and guard case. Below is a reimplementation of the switch statement we have been using:

  func add(_ topping: BurgerTopping) {
    if case let .Bun(type, seeds), seeds == true {
      print("\(type) bun with seeds. Yeeha!")
    }

    if case let .Bun(type, seeds), seeds == false {
      print("Ooooo. A lovely \(type) bun. Good choice!")
    }

    if case let .Patty(type) = topping {
      print("Juicy. Delicious. Succulant \(type) patty")
    }

    if case let .Cheese(type) = topping {
      print("You genius! \(type) Cheese. Sounds great!")
    }

    if case let .Pickle = topping {
      print("Pickles? Ok, if you're sure")
    }

    if case .Tomato = topping {
      print("Gotta love those tomatoes")
    }
  }

You may notice this is more verbose than the switch case scenario, due mostly to curly braces and newlines, but yields the exact same results. Writing if case let x = y { ... } is strictly equivalent to writing switch y { case let x: ... }:; it’s just a more compact syntax useful when you only want to match against one case as opposed to a switch which is more appropriate to multiple case matching.

Take note that the where clause here is represented by a single comma, creating a multi-clause conditional statement.

What Else?

Beyond the control flow constructs mentioned above, pattern matching can also be used with for case and types. I have not yet needed to use these features and may write about them in a future post. Until then I am going to continue to altering my development habits to accommodate this functional pattern.

External Resources

Patterns. See developer.apple.com.
Match Me if you can: Swift Pattern Matching in Detail. See appventure.me.

Swift 3 - Enums

2016-11-12T00:00:00+00:00

I am always looking out for better solutions to implement software features where the language lends itself well to the problem. This way I get the most out of the language and offload some responsibility to the specific nuances the language offers. In this article I talk about Swift enums and how I have used them.

What is an Enum?

An enum encapsulates a group of related values within a particular domain. Unlike enums in C which map values to a set of integers, Swift does not enforce this mapping. If however a value is associated with the related values it is not limited to integers, but can be represented by any type conforming to RawRepresentable protocol. This means that an associated value can a string, a character, or a value of any integer or floating-point type.

Swift extends enums further to adopt features traditionally supported only by classes. This includes the adoption of instance methods, computed properties, initialisers and the ability to conform to protocols. These features are particularly useful when needing to implement enums where associated values do no conform to RawRepresentable.

Using Enum To Implement a Type

Part of an app I am writing required a way to encapsulate the colour scheme I intend to use. Since there would be a discrete number of colours an enum seemed like the best solution. It was also important that I be able to select a random colour from the enum.

Using an enum the solution was simple as shown in this snippet:

  enum ColorPalette: UInt32 {
    case blue
    case green
    case orange
    case pink
    case purple
    case red
    case turquoise

    var color: UIColor {
      switch self {
      case .blue: return UIColor(red: 0.39, green: 0.62, blue: 0.71, alpha: 1)
      case .green: return UIColor(red: 0.77, green: 0.87, blue: 0.59, alpha: 1)
      case .orange: return UIColor(red: 0.96, green: 0.76, blue: 0.33, alpha: 1)
      case .pink: return UIColor(red: 0.98, green: 0.82, blue: 0.85, alpha: 1)
      case .purple: return UIColor(red: 0.56, green: 0.47, blue: 0.76, alpha: 1)
      case .red: return UIColor(red: 0.69, green: 0.36, blue: 0.46, alpha: 1)
      case .turquoise: return UIColor(red: 0.35, green: 0.78, blue: 0.72, alpha: 1)
      }
    }

    static func random() -> ColorPalette {
      let randomValue = arc4random_uniform(7)
      return ColorPalate(rawValue: randomValue)!
    }
  }

Take note of the implementation of random(). It instantiates and returns a ColourPalette using a raw value to identify one of it’s cases. Each case has been automatically assigned a numerical default value because the enum has been declared to store associated values of the UInt32 type.

Note: As stated in the Apple documentation arc4random_uniform() is recommended over constructions like arc4random() % upper_bound as it avoids modulo bias when the upper bound is not a power of two.

Starting out in CryEngine V

2016-10-26T00:00:00+00:00

It is an irrefutable fact that CryEngine is a fantastic game engine. This is made evident be the AAA titles that are being released. As a developer I have been keen to get in on this action and create my own stunning landscapes with a captivating storyline. However upon being faced with a blank canvas I really do feel like the David to this game engine Goliath.

This article clearly is not going to be an in depth discussion about how professionals use this engine and how to implement the next Crysis, however in this article you will learn the answers I had to some very basic question when starting out.

What is CryEngine V?

CryEngine V is the fifth major iteration of the CryEngine game engine developed by the German company Crytek. Since the release of the original engine in 2002 it has been used for all their titles and has been continually updated to support newer technologies, hardware and platforms. This brings us all the way to CryEngine V, released on 22 March 2016, to support a new licensing model allowing the engine to be used for free.

The implementation of a launcher, which is used to download the engine, is a much needed improvement. In the past when stating a game project a new copy of the engine would need to be downloaded and the game built around the engine source code. From talking to other game developers who used CryEngine back in these archaic days they said that they preferred that way of working. Although Crytek are not the first to do this I personally disagree with the other developers and am glad Crytek have taken this approach.

How does it differ from UE4 or other engines?

Before settling with CryEngine V I also ventured into UnrealEngine 4 (UE4) and Unity. Although all of these game engines offer similar features there were some key points that made CryEngine V stand out to me:

100% Royalty free. Unlike UE4 and Unity which require you to pay royalties when you hit the big time, CryEngine is completely royalty free,
Real time sandbox editing. Both UE4 and Unity have this functionality, but it felt more solid in CryEngine,
Single key press to jump straight into the game. This is facilitated without loading the game engine as it is already running within the editor,
A lot of boilerplate is already done for you. A new project will start with an ocean, a small platform, a skybox and Time of Day (TOD) feature with daylight/nighttime transition. This was not something I found in UE4,
An impressive sample level which I feel is more accessible than those on UE4 or Unity. For instance it runs on my mid spec Windows Desktop whereas the UE4 Infiltrator demo has no hope in hell of running on my machine.

What CryEngine is not

So far I have been singing CryEngine praises, but like most things it is not perfect. In particular:

It is not as user friendly as other engines, although it is vastly more so than in previous versions,
It currently has little support and scattered documentation,
I found there to be a partially elitist community (at least those I spoke to on Slack), whilst the forums are most helpful,
It’s asset store is lacking in… well, assets. I assume this will improve over time. CryEngine (with launcher and store) is very new so the community has yet to release many assets.

The Missing Answers

Now with an introduction to the game engine I present the questions that I had when starting to use CryEngine V.

1. How to Launch the Code, Editor and the Game?

I believe it is important to note that Visual Studio (or MonoDevelop for C#) should be launched using the Code_CPP.bat file (Code_CS.bat for C#) created after bootstrapping a new game project. This script creates a series of environment variables that presumably enable Visual Studio to successfully build the game.

Similarly launching the sandbox editor should be done via the Editor.bat file or from the CryEngine Launcher.

Update: Now CryEngine 5.2 has been released the above has been updated. Instead a working installation of CMake (3.6+) is required to generate a Visual Studio (or MonoDevelop for C#) project file that can be opened through explorer, whilst the editor and game are launched through the .cryproject file.

2. How are the Config Files Loaded?

There are a couple of configuration files scattered around the engine and individual projects. These are loaded in a particular order as discussed below:

system.cfg is the first config file loaded by the engine and found in the engine root. I tend not to touch this file and pretend it does not exist. Unless I know I want a certain configuration for every game I plan to create it will suffice to put my game configs elsewhere. The exception to this is the audio middleware. It was recommended that I switch middleware in here.

editor.cfg is loaded next and also found in the engine root. Presumably this config applies only to the editor. As with system.cfg, unless I know I want a certain configuration for every game I wont edit this file.

game.cfg is loaded next. This is the configuration file for the specific project.

Update: Prior to version 5.2 project.cfg is the loaded last. This file seemed superflous to me as game.cfg would do the job. It would appear it has been removed.

Upadte: Having updated to version 5.3 it appears that system.cfg has been removed so I was unable to continue using this file to host my audio middleware configurations. Instead I have moved it over to game.cfg. This has made a little more sense to me considering how the new plugin system is supposed to help separate engine logic from specific game code. Putting my configurations variables here keeps all my game specific values in one place.

The various configuration variables and console commands are documented on the CryEngine website and prefixed with the initials of it parent subcomponent of the engine. For example the s_ prefix is for the sound engine subcomponent. A full list of the available configuration variables can be found here.

3. What is the Resource Compiler?

The MakeAssets.bat invokes the the resource compiler (rc.exe) that optimises assets for the specific platform. Some useful resources referencing the resource compiler can be found on the CryEngine website. Ones of note are: using the resource compiler and compiling assets for multiple platforms.

Furthermore MakeAssets.bat contains syntax such as %~dp0. This is a Windows shell command that expands the full path to the file (including the drive letter drive). More on this can be found in this StackOverflow answer.

Update: In version 5.2 of the engine this feature seems to be broken since the project.cfg file was removed. I questioned this on the forums to only have it confirmed. Hopefully this will be fixed ASAP.

Update: In version 5.3 MakeAssets.bat has been removed from the game templates. I can only assume it is now handled entirely within the CMake system.

4. How to implement Wwise sound engine?

When I start a project I like to setup and integrate all the other 3rd party software I intend to use first. So to begin I wanted to switch the audio middleware that the engine uses. By default this is SDL Mixer. In order for the engine to interact with different audio middleware it implements an Audio Translation Layer (ATL) which is an abstraction interface between CryEngine and other audio middlewares i.e. Wwise.

I wont be going through the setup of Wwise in this article as it is covered well here but the two important console commands I used are:

s_DrawAudioDebug 1  <-- Display debug information about ATL middleware.
s_AudioImplName CryAudioImplWwise  <-- Switch to the Wwise audio middleware. Other options include CryAudioImplFmod (for fmod studio) and CryAudioImplSDLMixer (for sdl mixer).

I also set these in my system.cfg file as mentioned earlier. Now every time I create a new project, or launch a current project the correct audio middleware is selected.

Update: As of version 5.3 I have transitioned this configuration to game.cfg as system.cfg has been removed (at least according to my editor log file which is unable to find it).

5. How is the C++ Sample Project Structured?

I lean toward using C++ when I can. It seems to have been the industry standard for many years so there must be something about it that lends itself well to games. Alternatively it was just the only decent language available at the time and has just stuck. Regardless, the C++ project the launcher generates includes a little boilerplate that is not well documented and I simply am unsure what it all does, whether it is necessary and whether I can remove it without breaking everything.

I’m cautious that this is an example of functional fixedness, a bias limiting me to using the engine only in the way it has traditionally been used. So I intend to figure out what were the intentions of having included these particular boilerplate features, yet not implement others? Beyond this I intend to find what each bit does.

6. What is Included in the Assets Folder?

The Assets folder contains Textures.pak. What is the difference between this and the terraintexture.pak within Assets/levels/example? I assume Textures.pak are global textures whereas terraintexture.pak contains textures pertaining to a particular levels landscape including height maps, normal maps etc…

Moving Forward

I am clearly still no expert here, with some questions still unanswered. I fully intend to keep this article up to date whilst I discover more answers to my questions, perhaps adding further questions as they develop.

Please leave comments correcting any of my assumptions as I have likely made many mistakes in my findings. I would like this to turn into a great starting out guide to those jumping in at the deep end like myself.

Building a Bar Top Multicade

2016-08-29T00:00:00+00:00

For sometime I have wanted an arcade machine for retro gaming, but they can be extremely expensive and take up a lot of space. This is no more true then a full sized cabinet. Furthermore they often come preloaded with only a handful of games that become tiresome pretty quickly. So I set out to solve these problems, saving space and achieving an affordable cabinet capable of playing hundreds of old, classic games.

Of course I am not the first, and I most certainly wont be the last to make my own arcade cabinet, but it must be said the high level of customisation is satisfying and rewarding in the end.

Overall building my own cabinet has taken just over a year, mostly due to finding and waiting on joinery companies to cut out my custom wooden panels as I possess neither the steady hand, patience or machinery to do it myself. If Dr. Leonard McCoy worked in IT I’m sure he’d say something along the lines of “Im a software engineer, not a machinist”.

My arcade is a DIY retro bartop arcade cabinet built for two players whom can use either the built in joysticks and buttons or plug in SNES controllers for that added retro console feel. It is powered by a Raspberry Pi Model B capable of playing multiple types of retro games - primarily NES, SNES, Megadrive and arcade (MAME) games.

Materials List

I believe in total the arcade has cost me around £250, but considering that is a fraction of the cost of some machines it is a real steal. The bill of materials I used to construct the arcade follows:

Item	Qty.	Merchant
Raspberry Pi Model B (or better)	1	Pimoroni
Heatsink for Raspberry Pi	1	Amazon
32 GB SD Card	1	Amazon
3 Pin Inlet	1	Amazon
Kettle Lead	1	Amazon
4-Way Extension	1	Amazon
SNES Controller Extension	1	Amazon
SNES Controller	2	eBay
19” 5:4 TFT Monitor	1	eBay
Joysticks, Buttons and Controller	1	Arcade World UK
Speakers and Amplifier	1	Arcade World UK
T Moulding (1m)	5	Arcade World UK
19mm MDF Panels		B&Q

Note: Heatsink is not necessary on newer Raspberry Pi models.