A deep dive into the practical challenges of implementing, testing, and maintaining Universal Links at scale

Universal Links have been around since iOS 9 (2015), yet the topic remains surprisingly underrated in the iOS community. While most developers understand the basic concept (associating your website with your app so links open directly in the app) the practical challenges of implementing and maintaining Universal Links at scale are rarely discussed.

When a user taps a Universal Link, iOS checks if the domain is associated with any installed app. If it is, iOS opens the app directly. If not, it opens the link in the browser. This "universal" behavior makes them superior to custom URL schemes (deep links) for user-facing communications.

However, despite their importance, many developers treat AASA files as simple configuration files, overlooking the complex challenges involved in validating, testing, and maintaining them at scale.

In 2024, I put a lot of effort into crafting a solid solution for some overlooked challenges surrounding universal links. Every time I refer back to that work, I am impressed by how well it has served the company, which constantly renews my desire to write about it. GenAI has become incredibly helpful with the drafting process, so I finally have no excuse not to share this story!

In this post, I'll walk through the real-world challenges I've encountered and the solutions I've developed over the years working with Universal Links across multiple web domains and localized applications.

The Basics: What Makes Universal Links Work?

Universal Links work through a combination of three pieces:

1. Associated Domains Entitlement in your app (the .entitlements file)
2. Apple App Site Association (AASA) file on your website
3. Proper handling of incoming links in your app code

The AASA file must be served from /.well-known/apple-app-site-association over HTTPS, without redirects, and with the correct content type (application/json).

Here's a simple AASA file:

{
  "applinks": {
    "details": [
      {
        "appIDs": ["TEAMID.com.example.app"],
        "components": [
          { "/": "/account/login" },
          { "/": "/restaurants/*" }
        ]
      }
    ]
  }
}

What most tutorials don't tell you is that this is just the beginning. Real-world AASA files are far more complex, and validating them is a challenge in itself. The reality gets complicated when you need to:

• Validate that your AASA file respects a schema
• Test links before deploying to production
• Handle dynamic URL patterns with substitution variables
• Ensure Apple's CDN has picked up your latest changes
• Parse and match wildcard patterns correctly
• Handle encoding and special characters

Challenge 1: Nobody Validates Against a JSON Schema

Here's a dirty secret: most AASA files in production have never been validated against a schema. Teams deploy files, hope for the best, and only discover issues when links stop working.

Why It Matters: An invalid AASA file might be served successfully but fail to associate your app with your website. iOS won't throw errors; Universal Links simply won't work, and you might not notice until users report issues.

Online validators like branch.io/resources/aasa-validator and getuniversal.link check basic accessibility and JSON parsing, but they don't validate the actual schema. A file can be valid JSON yet completely invalid as an AASA file.

The Solution: JSON Schema Validation in CI

Create a comprehensive JSON Schema that validates the entire AASA structure, including:

• Required fields (applinks, details, appIDs, components)
• Optional fields (substitutionVariables, exclude, caseSensitive, percentEncoded)
• Proper nesting and data types
• Support for other AASA features (webcredentials, appclips, activitycontinuation)

Here's a schema that defines the correct structure of an AASA file (just for the applinks section) :

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "applinks": {
      "type": "object",
      "properties": {
        "defaults": {
          "type": "object",
          "properties": {
            "caseSensitive": {
              "type": "boolean"
            },
            "percentEncoded": {
              "type": "boolean"
            }
          }
        },
        "details": {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "appIDs": { "type": "array", "items": { "type": "string" } },
              "components": {
                "type": "array",
                "items": {
                  "type": "object",
                  "properties": {
                    "/": { "type": "string" },
                    "?": { "type": "object" },
                    "#": { "type": "string" },
                    "exclude": { "type": "boolean" },
                    "caseSensitive": { "type": "boolean" },
                    "percentEncoded": { "type": "boolean" }
                  },
                  "required": ["/"]
                }
              },
              "defaults": {
                "type": "object",
                "properties": {
                  "caseSensitive": { "type": "boolean" },
                  "percentEncoded": { "type": "boolean" }
                }
              }
            }
          }
        },
        "substitutionVariables": { "type": "object" }
      },
      "required": ["details"]
    }
  },
  "required": ["applinks"]
}

Integrating schema validation into your CI pipeline ensures invalid files never reach production. This catches issues like:

• Missing required fields
• Wrong types (string instead of array)
• Typos in property names (which would be silently ignored)
• Invalid component structures

You might want to consider building a Swift CLI tool with Argument Parser, in which case I would suggest using JSONSchema.swift.

Challenge 2: The Apple CDN Layer

Here's something that surprises many developers: iOS doesn't fetch the AASA file directly from your website. Instead, Apple operates a CDN that caches AASA files from websites.

The CDN URL follows this pattern:

https://app-site-association.cdn-apple.com/a/v1/<domain>

For example, for just-eat.co.uk:

• Website: https://just-eat.co.uk/.well-known/apple-app-site-association
• Apple CDN: https://app-site-association.cdn-apple.com/a/v1/just-eat.co.uk

Why It Matters: This caching happens periodically (every few hours), and there's no guarantee that your latest changes are immediately available. If your website's AASA file differs from what's cached on Apple's CDN, Universal Links may not work as expected. You might deploy a fix, but iOS devices could still be using the old cached version for hours or even days.

The Solution: CDN Validation

To ensure your AASA file has propagated correctly, you need to compare the file on your website with the one on Apple's CDN. This validates that:

1. Your file is publicly accessible and has a valid SSL certificate
2. The file has the correct MIME type (application/json)
3. Apple's CDN has successfully cached your latest version

Here's a validator that does exactly this:

struct AASAContent: Equatable, Decodable {
    let appLinks: AppLinks
    
    enum CodingKeys: String, CodingKey {
        case appLinks = "applinks"
    }

    // and nested Decodable structs
}

enum AASAFileLocation {
    case website
    case appleCdn

    func buildURL(with domain: Domain) throws -> URL {
        switch self {
        case .website:
            return URL(string: "https://\(domain)")!
                .appendingPathComponent(".well-known")
                .appendingPathComponent("apple-app-site-association")
        case .appleCdn:
            return URL(string: "https://app-site-association.cdn-apple.com/a/v1/")!
                .appendingPathComponent(domain)
        }
    }
}
    
func validateCDN(for domain: Domain) async throws {
    let websiteURL = try AASAFileLocation.website.buildURL(with: domain)
    let appleCdnURL = try AASAFileLocation.appleCdn.buildURL(with: domain)

    let domainFile: AASAContent = try await downloadFile(url: websiteURL)
    let appleFile: AASAContent = try await downloadFile(url: appleCdnURL)

    guard domainFile == appleFile else {
        throw ValidateCDNError.fileMismatch(domain: domain)
    }
}

Running this validation daily in CI ensures you're alerted when CDN synchronization fails or is delayed. A daily automated check can alert you if there's a mismatch, allowing you to investigate and resolve issues before they impact users. This simple check has prevented numerous incidents where teams assumed links were working when they weren't.

Developer Mode Bypass

For development and debugging, iOS offers a bypass. By adding ?mode=developer to your associated domain:

<string>applinks:just-eat.co.uk?mode=developer</string>

Debug builds should use a specific entitlements file where the developer mode is used. Debug builds will fetch the AASA file directly from your domain, bypassing the CDN. This requires enabling "Associated Domains Development" in iOS Settings → Developer. App Store builds always use the CDN and their entitlements file shouldn't mention the developer mode.

Challenge 3: Regular Expression Parsing and Pattern Matching

The AASA file supports powerful pattern matching through wildcards for flexible URL matching. However, these patterns aren't standard regex and use Apple's own pattern syntax that needs to be converted to regular expressions for validation.

The Pattern Syntax:

• * matches zero or more characters (converted to .* in regex)
• ? matches exactly one character (converted to . in regex)
• ?* matches one or more characters (converted to .+ in regex)
• *? also matches one or more characters (converted to .+ in regex)

The Problem: I couldn't find any online tool or open-source library implementing Apple's matching logic. If you want to validate that specific URLs match your AASA file patterns (for testing or regression prevention), you need to correctly parse and convert these patterns. Online validators like Branch.io's AASA validator don't support this matching logic, they only validate the file structure.

The Solution: Implementing Apple's Matching Logic

I built a custom validator that implements the matching rules. The key insight is that Apple's wildcards map to regular expressions. A naïve conversion (where the order of substitutions is important) would look like this:

extension String {
    var regEx: String {
        self
            // One or more characters
            .replacingOccurrences(of: "?*", with: ".+")
            
            // One or more characters
            .replacingOccurrences(of: "*?", with: ".+")
            
            // Zero or more characters
            .replacingOccurrences(of: "*", with: ".*")
            
            // Exactly one character
            .replacingOccurrences(of: "?", with: ".")
    }
}

Additionally, you need to handle URL components properly. For example, if a pattern specifies only a path (/restaurants/*), you should still match URLs that have query parameters or fragments, unless explicitly excluded. This requires careful construction of the regex pattern to account for optional components, which to be completely honest was very tricky to implement by hand at a time when LLMs weren't too helpful.

Challenge 4: The substitutionVariables Problem

Apple supports applinks.substitutionVariables for dynamic URL matching. This feature allows you to define variables that can be used in path, query, and fragment components. Substitution variables are particularly helpful to reduce duplication when dealing with URLs that are localised per language. However, I couldn't find any online validator or open-source tool to support validating links against AASA files that use substitution variables.

Here's a real-world example from a multi-language website:

{
  "applinks": {
    "substitutionVariables": {
      "menu": ["speisekarte", "menu"],
      "stamp-cards": ["stempelkarten", "stamp-cards", "cartes-épargne", "stempelkaarten"]
    },
    "details": [{
      "appIDs": ["TEAMID.com.example.app"],
      "components": [
        { "/": "/$(lang)/$(menu)/?*" },
        { "/": "/", "#": "$(stamp-cards)" },
        { "/": "/", "#": "$(order-history)" }
      ]
    }]
  }
}

The pattern /$(lang)/$(menu)/?* should match URLs like:

• /de/speisekarte/restaurant-name
• /en/menu/restaurant-name
• /fr/menu/pizza-place

And /#$(stamp-cards) should match:

• /#stempelkarten
• /#stamp-cards
• /#cartes-épargne

Why It Matters: Without proper support for substitution variables, you can't validate that your Universal Links work correctly. You might think a URL should match, but if the substitution isn't handled correctly, it won't.

The Solution: Substitution Variable Expansion

Implement substitution variable expansion before pattern matching. Here is a trimmed down example:

1. Parse substitution variables from the AASA file
2. Replace variable references ($(variableName)) with regex alternatives of their possible values
3. Handle default variables like $(lang) and $(region) which match any two characters
4. Apply the expanded pattern to URL matching after converting Apple's pattern syntax to standard regex

The key insight is that substitution variables create a disjunction (OR) of possible values:

func replaceWithSubstitutionVariables(_ substitutionVariables: [String: [String]]) -> String {
    var modifiedString = self
    let substitutionVariablesWithDefaults = substitutionVariables.merging(defaultSubstitutionVariables) { (current, _) in current }
    
    for (key, values) in substitutionVariablesWithDefaults {
        let pattern = "\\$\\(\(key)\\)"
        let replacement = "(\(values.joined(separator: "|")))"
        // Replace $(key) with (value1|value2|value3)
        modifiedString = modifiedString.replacingOccurrences(
            of: pattern, 
            with: replacement, 
            options: .regularExpression
        )
    }
    return modifiedString
}

private var defaultSubstitutionVariables: [String: [String]] {
    [
        "lang": [".."],
        "region": [".."]
    ]
}

So /$(lang)/$(menu)/?* with the variables above becomes:

/(..)/((speisekarte|menu))/.+

Note: $(lang) and $(region) are special default variables Apple provides that match any two characters.

The order of operations matters: first expand substitution variables, then convert Apple's pattern syntax (*, ?, ?*) to standard regex. This ensures that wildcards within substitution variable values are handled correctly.

The Full Matching Pipeline

The complete validation process:

1. Parse the AASA file and extract components for the target bundle ID
2. For each component, build a regex pattern by:
   • Replacing substitution variables with alternations
   • Converting * and ? to regex equivalents
   • Handling paths, query parameters, and fragments
3. Match incoming URLs against these patterns
4. Account for the exclude flag that explicitly prevents matching

The matcher also needs to handle edge cases:

• URLs with query parameters not specified in the component (allowed)
• URLs with fragments not specified in the component (allowed)
• The exclude: true flag that creates negative matches
• Case sensitivity settings
• Percent encoding

Here's the core matching logic:

enum AllowPolicy {
    case allowed
    case notAllowed
}

func validateDeepLinking(
    policy: AllowPolicy,
    for url: URL,
    domain: Domain,
    components: [AASAContent.AppLinks.Detail.Component],
    substitutionVariables: AASAContent.AppLinks.SubstitutionVariables
) throws {
    switch policy {
    case .allowed:
        for component in components {
            let regEx = try regEx(for: component, substitutionVariables: substitutionVariables, on: domain)
            if findMatch(for: url, in: regEx) {
                if component.exclude != true {
                    return
                } else {
                    throw ValidateUniversalLinkError.excludedUniversalLink(url: url)
                }
            }
        }
        throw ValidateUniversalLinkError.unhandledUniversalLink(url: url)
    case .notAllowed:
        for component in components {
            let regEx = try regEx(for: component, substitutionVariables: substitutionVariables, on: domain)
            if findMatch(for: url, in: regEx) {
                if component.exclude == true {
                    return
                } else {
                    throw ValidateUniversalLinkError.incorrectlyHandledUniversalLink(url: url)
                }
            }
        }
    }
}

Important: Components are evaluated in order, and the first match wins. This means exclusion rules must come before the broader patterns they're excluding from.

Challenge 5: Testing Before Production

One of the trickiest aspects of Universal Links is testing. You can't just deploy to production and hope it works. Testing requires the AASA file to be hosted on a real domain with proper SSL certificates. You can't just test locally or in a simulator without additional setup.

Why It Matters: Deploying untested AASA changes to production can break Universal Links for all users. Since Apple caches AASA files, fixing issues can take hours or days to propagate.

The Solution: A Staging Environment with Real Domains

Set up a staging environment using AWS infrastructure (or similar):

1. S3 bucket to host AASA files
2. CloudFront distributions for each staging domain with HTTPS
3. Route53 records pointing staging subdomains to CloudFront

The staging domains follow a pattern like:

lieferando-de.aasa-staging.mobile-team.example.com

This mirrors the production domain lieferando.de and serves the same AASA file structure.

Debug builds include both staging and production domains in its entitlements:

<key>com.apple.developer.associated-domains</key>
<array>
    <string>applinks:lieferando-de.aasa-staging.mobile-team.example.com</string>
    <string>applinks:lieferando.de</string>
    <string>applinks:www.lieferando.de</string>
</array>

Now you can test a link like:

https://lieferando-de.aasa-staging.mobile-team.example.com/menu/pizzeria

Instead of:

https://lieferando.de/menu/pizzeria

The staging URL opens your debug/ad-hoc build exactly like the production URL would open your App Store build, but without risking production changes.

Important Testing Considerations

• TestFlight builds are production builds and therefore use production entitlements and won't work with staging domains
• Ad-hoc and debug builds can include staging domains
• Simulator testing has limitations. Universal Links work best on physical devices
• Developer mode must be enabled on device for direct AASA fetching (bypassing CDN)
• For non-debug builds, you'll need to wait for Apple's CDN to cache your staging AASA file

Challenge 6: Regression Testing and Comprehensive Link Validation

Manual testing doesn't scale. Every change to the AASA file needs verification across dozens or hundreds of URLs. And you need to verify both:

1. URLs that should open the app (deep-linkable)
2. URLs that should not open the app (non-deep-linkable, excluded)
3. Complex URLs with query parameters, fragments, and wildcards

Why It Matters: Without comprehensive validation, you might:

• Miss URLs that should deep link but don't
• Accidentally deep link URLs that should open in the browser
• Break existing functionality when making changes

The Solution: Automated Content Validation

I maintain JSON files alongside each AASA file listing the expected behavior:

{
  "deep_linkable_urls": [
    "https://lieferando.de/",
    "https://lieferando.de/punkte",
    "https://lieferando.de/#stempelkarten",
    "https://lieferando.de/en#stamp-cards",
    "https://lieferando.de/lieferservice/essen/berlin-10115",
    "https://lieferando.de/en/delivery/food/berlin-10115"
  ],
  "non_deep_linkable_urls": [
    "https://lieferando.de/?openOnWeb=true",
    "https://lieferando.de/anyPath?openOnWeb=true"
  ]
}

The validator ensures:

• Every URL in deep_linkable_urls matches a non-excluded component
• Every URL in non_deep_linkable_urls either doesn't match or matches an excluded component

This runs on CI on every pull request. Changes to the AASA file must include updates to the expected URLs, creating living documentation of what's supported.

Error Types

The validator catches several error conditions:

1. Unhandled URL: A URL expected to be deep-linkable doesn't match any component
2. Excluded URL: A URL expected to be deep-linkable matches an excluded component
3. Incorrectly Handled URL: A URL expected to be non-deep-linkable actually matches a component

Each error provides clear diagnostics:

enum ValidateUniversalLinkError: Error {
    case unhandledUniversalLink(url: URL)
    case excludedUniversalLink(url: URL)
    case incorrectlyHandledUniversalLink(url: URL)
    // ...
}

Challenge 7: Encoding and Special Characters

Universal Links often contain special characters, especially for localized content:

https://lieferando.at/fr#cartes-épargne

The fragment cartes-épargne contains an accented character. Here's the critical rule:

Universal Links must NOT be percent-encoded in the AASA file or when shared with users.

✅ Correct: https://lieferando.at/fr#cartes-épargne
❌ Wrong: https://lieferando.at/fr#cartes-%C3%A9pargne

However, when these URLs are processed by URLComponents in Swift, they may get encoded. The validator must handle both forms and compare them correctly:

private func findMatch(for url: URL, in regEx: NSRegularExpression) -> Bool {
    let searchString = url.absoluteString.removingPercentEncoding!
    let searchRange = NSRange(location: 0, length: searchString.utf16.count)
    if let result = regEx.firstMatch(in: searchString, options: [.anchored], range: searchRange) {
        return result.range.length == searchRange.length
    }
    return false
}

The key is to decode the URL before matching against the regex.

Putting It All Together: The Complete Validation Pipeline

To address all these challenges, I built AASAValidator, a Swift command-line tool that provides three main commands:

1. validate-schema: Validates AASA files against a JSON schema
2. validate-cdn: Compares your website's AASA file with Apple's CDN version
3. validate-universal-links: Validates that specific URLs match (or don't match) your AASA file patterns

The tool handles:

• JSON schema validation
• CDN comparison
• Regular expression parsing and conversion
• Substitution variable expansion
• Complex URL matching with query parameters and fragments
• Exclusion validation
• Percent encoding

The Full Automation Pipeline

1. PR Validation (CI)

• Schema validation: Ensure AASA files are structurally correct
• Content validation: Verify all expected URLs match (or don't match) correctly
• Bundle ID validation: Ensure the target app's bundle ID is in the AASA

2. Post-Deployment (Staging)

• Deploy AASA files to staging environment
• Test Universal Links on physical devices with staging builds
• Verify end-to-end behavior

3. Post-Deployment (Production)

• Deploy AASA files to production
• CDN validation: Check that Apple's CDN has the latest version
• Smoke test with production app

4. Ongoing Monitoring

• Daily CDN synchronization checks
• Alerting if files fall out of sync

Example GitHub Actions Integration

Here's how you might use the validator in a GitHub Actions workflow:

strategy:
  fail-fast: false
    matrix:
      domain:
        - just-eat.co.uk
        - just-eat.es
      bundle-id:
        - com.eatch.mobileapp
        - com.justeat.JUSTEAT
        - com.takeaway.lu

steps:

  - name: Validate AASA Schema
    run: |
      AASAValidator validate-schema \
        --aasa-path ./${{ matrix.domain }}-aasa.json \
        --json-schema-path path/to/JSONSchema.json

  - name: Validate Universal Links
    run: |
      AASAValidator validate-universal-links \
        --bundle-id ${{ matrix.bundle-id }} \
        --domain ${{ matrix.domain }} \
        --aasa-path ./${{ matrix.domain }}-aasa.json \
        --universal-links-path ./universal-links/${{ matrix.domain }}.json

Key Takeaways

1. Validate against a schema: JSON parsing success doesn't mean your AASA file is valid. Use JSON Schema validation in CI to catch structural errors early.
2. Don't trust the CDN blindly: Always verify that Apple's CDN has your latest AASA file. Implement automated daily checks.
3. Implement custom regex parsing: No existing tool handles Apple's wildcard pattern syntax. Build your own matcher to validate URL matching.
4. Test substitution variables properly: No existing tool handles applinks.substitutionVariables. Build your own matcher or use custom validation logic.
5. Implement proper staging: Set up real staging domains with HTTPS. Testing Universal Links requires real infrastructure.
6. Automate regression testing: Maintain lists of expected URLs and validate them automatically. Manual testing doesn't scale.
7. Watch your encoding: Special characters must not be percent-encoded in Universal Links. Handle encoding carefully in your validation logic.
8. Order matters for exclusions: Components are evaluated in order. Place exclusion rules before broader patterns.
9. Plan for multi-language support: If your website supports multiple languages, your AASA file needs substitution variables to handle localized URLs.

Conclusion

Universal Links are deceptively simple on the surface but require careful attention to detail in practice. The challenges we've discussed (schema validation, CDN synchronization, regex parsing, substitution variables, staging environments, comprehensive link validation, and encoding) are often ignored but are essential for a robust implementation.

By building tooling that addresses these challenges and integrating validation into your development workflow, you can ensure that Universal Links work reliably for your users. The investment in proper validation and testing pays off by preventing production issues and giving you confidence when making changes to your AASA files.

Consider implementing similar validation for your own Universal Links setup and your future self will thank you.

Based on this article, I've published Luca!
A lightweight decentralised tool manager for macOS to manage project-specific tool environments.

👶 Check it out: luca.tools

Overview

It's common for iOS teams to rely on various CLI tools such as SwiftLint, Tuist, and Fastlane. These tools are often installed in different ways. The most common way is to use Homebrew, which is known to lack version pinning and, as Pedro puts it:

Homebrew is not able to install and activate multiple versions of the same tool

I also fundamentally dislike the tap system for installing dependencies from third-party repositories. Although I don't have concrete data, I feel that most development teams profoundly dislike Homebrew when used beyond the simple installation of individual tools from the command line and the brew taps system is cumbersome and bizarre enough to often discourage developers from using it.

Alternatives to manage sets of CLI tools that got traction in the past couple of years are Mint and Mise. As Pedro again says in his article about Mise:

The first and most core feature of Mise is the ability to install and activate dev tools. Note that we say "activate" because, unlike Homebrew, Mise differentiates between installing a tool and making a specific version of it available. 

While beyond the scope of this article, I recommend a great article about installing Swift executables from source with Mise by Natan Rolnik.

In this article I describe a CLI tool manager very similar to what I've implemented for my team. I'll simply call it "ToolManager". The tool is designed to:

1. Support installing any external CLI tool distributed in zip archives
2. Support activating specific versions per project
3. Be decentralised (requiring no registry)

I believe the decentralisation is an interesting aspect and makes the tool reusable in any development environment. Also, differently from the design of mise and mint, ToolManager doesn't build from source and rather relies on pre-built executables.

In the age of GenAI, it's more important than ever to develop critical thinking and learn how to solve problems. For this reason, I won't show the implementation of ToolManager, as it's more important to understand how it's meant to work. The code you'll see in this article supports the overarching design, not the nitty-gritty details of how ToolManager's commands are implemented.

If, by the end of the article, you understand how the system should work and are interested in implementing it (perhaps using GenAI), you should be able to convert the design to code fairly easily—hopefully, without losing the joy of coding.

I myself am considering implementing ToolManager as an open source project later, as I believe it might be very helpful to many teams, just as its incarnation was (and continues to be) for the platform team at JET. There doesn't seem to be an existing tool with the design described in this article.

A different title could have reasonably placed this article in "The easiest X" "series" (1, 2, 3, 4), if I may say so.

Design

The point here is to learn what implementing a tool manager entails. I'll therefore describe the MVP of ToolManager, leaving out details that would make the design too straightforward to implement.

The tool itself is a CLI and it's reasonably implemented in Swift using ArgumentParser like all modern Swift CLI tools are.

In its simplest form, ToolManager exposes 3 commands:

• install:
  • download and installs the tools defined in a spec file (Toolfile.yml) at ~/.toolManager/tools optionally validating the checksum
  • creates symlinks to the installed versions at $(PWD)/.toolManager/active
• uninstall:
  • clears the entire or partial content of ~/.toolManager/tools 
• clears the content of $(PWD)/.toolManager/active
• version:
  • returns the version of the tool

The install commands allows to specify the location of the spec file using the --spec flag, which defaults to Toolfile.yml in the current directory.

The installation of ToolManager should be done in the most raw way, i.e. via a remote script. It'd be quite laughable to rely on Brew, wouldn't it?

This practice is commonly used by a variety of tools, for example originally by Tuist (before the introduction of Mise) and... you guessed it... by Brew. We'll see below a basic script to achieve so that you could host on something lik AWS S3 with the desired public permissions.

The installation command would be:

curl -Ls 'https://my-bucket.s3.eu-west-1.amazonaws.com/install_toolmanager.sh' | bash

The version of ToolManager must be defined in the .toolmanager-version file in order for the installation script of the repo to work:

echo "1.2.0" > .toolmanager-version

ToolManager manages versions of CLI tools but it's not in the business of managing its own versions. Back in the day, Tuist used to use tuistenv to solve this problem. I simply avoid it and have single version of ToolManager available at /usr/local/bin/ that the installation script overrides with the version defined for the project. The version command is used by the script to decide if a download is needed.

There will be only one version of ToolManager in the system at a given time, and that's absolutely OK.

At this point, it's time to show an example of installation script:

#!/bin/bash
set -euo pipefail

# Fail fast if essential commands are missing.
command -v curl >/dev/null || { echo "curl not found, please install it."; exit 1; }
command -v unzip >/dev/null || { echo "unzip not found, please install it."; exit 1; }

readonly EXEC_NAME="ToolManager"
readonly INSTALL_DIR="/usr/local/bin"
readonly EXEC_PATH="$INSTALL_DIR/$EXEC_NAME"
readonly HOOK_DIR="$HOME/.toolManager"
readonly REQUIRED_VERSION=$(cat .toolmanager-version)

# Exit if the version file is missing or empty.
if [[ -z "$REQUIRED_VERSION" ]]; then
  echo "Error: .toolmanager-version not found or is empty." >&2
  exit 1
fi

# Exit if the tool is already installed and up to date.
if [[ -f "$EXEC_PATH" ]] && [[ "$($EXEC_PATH version)" == "$REQUIRED_VERSION" ]]; then
  echo "$EXEC_NAME version $REQUIRED_VERSION is already installed."
  exit 0
fi

# Determine OS and the corresponding zip filename.
case "$(uname -s)" in
  Darwin) ZIP_FILENAME="$EXEC_NAME-macOS.zip" ;;
  Linux)  ZIP_FILENAME="$EXEC_NAME-Linux.zip" ;;
  *)      echo "Unsupported OS: $(uname -s)" >&2; exit 1 ;;
esac

# Download and install in a temporary directory.
TMP_DIR=$(mktemp -d)
trap 'rm -rf "$TMP_DIR"' EXIT # Ensure cleanup on script exit.

echo "Downloading $EXEC_NAME ($REQUIRED_VERSION)..."
DOWNLOAD_URL="https://github.com/MyOrg/$EXEC_NAME/releases/download/$REQUIRED_VERSION/$ZIP_FILENAME"
curl -LSsf --output "$TMP_DIR/$ZIP_FILENAME" "$DOWNLOAD_URL"
unzip -o -qq "$TMP_DIR/$ZIP_FILENAME" -d "$TMP_DIR"

# Use sudo only when the install directory is not writable.
SUDO_CMD=""
if [[ ! -w "$INSTALL_DIR" ]]; then
  SUDO_CMD="sudo"
fi

echo "Installing $EXEC_NAME to $INSTALL_DIR..."
$SUDO_CMD mkdir -p "$INSTALL_DIR"
$SUDO_CMD mv "$TMP_DIR/$EXEC_NAME" "$EXEC_PATH"
$SUDO_CMD chmod +x "$EXEC_PATH"

# Download and source the shell hook to complete installation.
echo "Installing shell hook..."
mkdir -p "$HOOK_DIR"
curl -LSsf --output "$HOOK_DIR/shell_hook.sh" "https://my-bucket.s3.eu-west-1.amazonaws.com/shell_hook.sh"
# shellcheck source=/dev/null
source "$HOOK_DIR/shell_hook.sh"

echo "Installation complete."

You might have noticed that:

• the required version of ToolManager (defined in .toolmanager-version) is downloaded from the release from the corresponding GitHub repository if missing locally. The ToolManager repo should have a GHA workflow in place to build, archive and upload the version.
• a shell_hook script is downloaded and run to insert the following line in the shell profile: [[ -s "$HOME/.toolManager/shell_hook.sh" ]] && source "$HOME/.toolManager/shell_hook.sh". This allows switching location in the terminal and loading the active tools for the current project.

Showing an example of shell_hook.sh is in order:

#!/bin/bash
# Overrides 'cd' to update PATH when entering a directory with a local tool setup.

# Add the project-specific bin directory to PATH if it exists.
update_tool_path() {
  local tool_bin_dir="$PWD/.toolManager/active"
  if [[ -d "$tool_bin_dir" ]]; then
    export PATH="$tool_bin_dir:$PATH"
  fi
}

# Redefine 'cd' to trigger the path update after changing directories.
cd() {
  builtin cd "$@" || return
  update_tool_path
}

# --- Installation Logic ---
# The following function only runs when this script is sourced by an installer.

install_hook() {
  local rc_file
  case "${SHELL##*/}" in
    bash) rc_file="$HOME/.bashrc" ;;
    zsh)  rc_file="$HOME/.zshrc" ;;
    *)
      echo "Unsupported shell for hook installation: $SHELL" >&2
      return 1
      ;;
  esac

  # The line to add to the shell's startup file.
  local hook_line="[[ -s \"$HOME/.toolManager/shell_hook.sh\" ]] && source \"$HOME/.toolManager/shell_hook.sh\""

  # Add the hook if it's not already present.
  if ! grep -Fxq "$hook_line" "$rc_file" &>/dev/null; then
    printf "\n%s\n" "$hook_line" >> "$rc_file"
    echo "Shell hook installed in $rc_file. Restart your shell to apply changes."
  fi
}

# This check ensures 'install_hook' only runs when sourced, not when executed.
if [[ "${BASH_SOURCE[0]}" != "$0" ]]; then
  install_hook
fi

Now that we have a working installation of ToolManager, let define our Toolfile.yml in our project folder:

---
tools:
  - name: PackageGenerator
    binaryPath: PackageGenerator
    version: 3.3.0
    zipUrl: https://github.com/justeattakeaway/PackageGenerator/releases/download/3.3.0/PackageGenerator-macOS.zip
  - name: SwiftLint
    binaryPath: swiftlint
    version: 0.57.0
    zipUrl: https://github.com/realm/SwiftLint/releases/download/0.58.2/portable_swiftlint.zip
  - name: ToggleGen
    binaryPath: ToggleGen
    version: 1.0.0
    zipUrl: https://github.com/TogglesPlatform/ToggleGen/releases/download/1.0.0/ToggleGen-macOS-universal-binary.zip
  - name: Tuist
    binaryPath: tuist
    version: 4.48.0
    zipUrl: https://github.com/tuist/tuist/releases/download/4.54.3/tuist.zip
  - name: Sourcery
    binaryPath: bin/sourcery
    version: 2.2.5
    zipUrl: https://github.com/krzysztofzablocki/Sourcery/releases/download/2.2.5/sourcery-2.2.5.zip

The install command of ToolManager loads the Toolfile at the root of the repo and for each defined dependency, performs the following:

• checks if the version of the dependency already exists on the machine
• if it doesn’t exist, downloads it, unzips it, and places the binary at ~/.toolManager/tools/ (e.g. ~/.toolManager/tools/PackageGenerator/3.3.0/PackageGenerator)
• creates a symlink to the binary in the project directory from .toolManager/active (e.g. .toolManager/active/PackageGenerator)

After running ToolManager install (or ToolManager install --spec=Toolfile.yml), ToolManager should produce the following structure

~ tree ~/.toolManager/tools -L 2
├── PackageGenerator
│   └── 3.3.0
├── Sourcery
│   └── 2.2.5
├── SwiftLint
│   └── 0.57.0
├── ToggleGen
│   └── 1.0.0
└── Tuist
    └── 4.48.0

and from the project folder

ls -la .toolManager/active
<redacted> PackageGenerator -> /Users/alberto/.toolManager/tools/PackageGenerator/3.3.0/PackageGenerator
<redacted> Sourcery -> /Users/alberto/.toolManager/tools/Sourcery/2.2.5/Sourcery
<redacted> SwiftLint -> /Users/alberto/.toolManager/tools/SwiftLint/0.57.0/SwiftLint
<redacted> ToggleGen -> /Users/alberto/.toolManager/tools/ToggleGen/1.0.0/ToggleGen
<redacted> Tuist -> /Users/alberto/.toolManager/tools/Tuist/4.48.0/Tuist

Bumping the versions of some tools in the Toolfile, for example SwiftLint and Tuist, and re-running the install command, should result in the following:

~ tree ~/.toolManager/tools -L 2
├── PackageGenerator
│   └── 3.3.0
├── Sourcery
│   └── 2.2.5
├── SwiftLint
│   ├── 0.57.0
│   └── 0.58.2
├── ToggleGen
│   └── 1.0.0
└── Tuist
    ├── 4.48.0
    └── 4.54.3

ls -la .toolManager/active
<redacted> PackageGenerator -> /Users/alberto/.toolManager/tools/PackageGenerator/3.3.0/PackageGenerator
<redacted> Sourcery -> /Users/alberto/.toolManager/tools/Sourcery/2.2.5/Sourcery
<redacted> SwiftLint -> /Users/alberto/.toolManager/tools/SwiftLint/0.58.2/SwiftLint
<redacted> ToggleGen -> /Users/alberto/.toolManager/tools/ToggleGen/1.0.0/ToggleGen
<redacted> Tuist -> /Users/alberto/.toolManager/tools/Tuist/4.54.3/Tuist

CI Setup

On CI, the setup is quite simple. It involves 2 steps:

• install ToolManager
• install the tools

The commands can be wrapped in GitHub composite actions:

name: Install ToolManager

runs:
  using: composite
  steps:
    - name: Install ToolManager
      shell: bash
      run: curl -Ls 'https://my-bucket.s3.eu-west-1.amazonaws.com/install_toolmanager.sh' | bash

name: Install tools

inputs:
  spec:
    description: The name of the ToolManager spec file
    required: false
    default: Toolfile.yml

runs:
  using: composite
  steps:
    - name: Install tools
      shell: bash
      run: |
        ToolManager install --spec=${{ inputs.spec }}
        echo "$PWD/.toolManager/active" >> $GITHUB_PATH

simply used in workflows:

- name: Install ToolManager
  uses: ./.github/actions/install-toolmanager

- name: Install tools
  uses: ./.github/actions/install-tools
  with:
    spec: Toolfile.yml

CLI tools conformance

ToolManager can install tools that are made available in zip files, without the need of implementing any particular spec. Depending on the CLI tool, the executable can be at the root of the zip archive or in a subfolder. Sourcery for example places the executable in the bin folder.

- name: Sourcery
  binaryPath: bin/sourcery
  version: 2.2.5
  zipUrl: https://github.com/krzysztofzablocki/Sourcery/releases/download/2.2.5/sourcery-2.2.5.zip

GitHub releases are great to host releases as zip files and that's all we need. Ideally, one should decorate the repositories with appropriate release workflows.

Following is a simple example that builds a macOS binary. It could be extended to also create a Linux binary.

name: Publish Release

on:
  push:
    tags:
      - '*'

env:
  CLI_NAME: my-awesome-cli-tool

permissions:
  contents: write

jobs:
  build-and-archive:
    name: Build and Archive macOS Binary
    
    runs-on: macos-latest
    
    steps:
      - name: Checkout repository
        uses: actions/checkout@v4

      - name: Setup Xcode
        uses: maxim-lobanov/setup-xcode@v1
        with:
          xcode-version: '16.4'

      - name: Build universal binary
        run: swift build -c release --arch arm64 --arch x86_64
      
      - name: Archive the binary
        run: |
          cd .build/apple/Products/Release/
          zip -r "${{ env.CLI_NAME }}-macOS.zip" "${{ env.CLI_NAME }}"
      
      - name: Upload artifact for release
        uses: actions/upload-artifact@v4
        with:
          name: cli-artifact
          path: .build/apple/Products/Release/${{ env.CLI_NAME }}-macOS.zip

  create-release:
    name: Create GitHub Release
    
    needs: [build-and-archive]
    
    runs-on: ubuntu-latest
    
    steps:
      - name: Download CLI artifact
        uses: actions/download-artifact@v4
        with:
          name: cli-artifact

      - name: Create Release and Upload Asset
        uses: softprops/action-gh-release@v2
        with:
          files: "${{ env.CLI_NAME }}-macOS.zip"

A note on version pinning

Dependency management systems tend to use a lock file (like Package.resolved in Swift Package manager, Podfile.lock in the old days of CocoaPods, yarn.lock/package-lock.json in JavaScript, etc.).

The benefits of using a lock file are mainly 2:

1. Reproducibility
   It locks the exact versions (including transitive dependencies) so that every team member, CI server, or production environment installs the same versions.
2. Faster installs
   Dependency managers can skip version resolution if a lock file is present, using it directly to fetch the exact versions, improving speed.

We can remove the need for lock files if we pin the versions in the spec (the file defining the tools). If version range operators like the CocoaPods' optimistic operator ~> and the SPM's .upToNextMajor and similar one didn't exist, usages of lock files would lose its utility.

While useful, lock files are generally annoying and can create that odd feeling of seeing unexpected updates in pull requests made by others. ToolManager doesn't use a lock file; instead, it requires teams to pin their tools' versions, which I strongly believe is a good practice.

This approach comes at the cost of teams having to keep an eye out for patch releases and not leaving updates to the machine, which risks pulling in dependencies that don't respect Semantic Versioning (SemVer).

Support for different architectures

This design allows to support different architectures. Some CI workflows might only need a Linux runner to reduce the burden on precious macOS instances. Both macOS and Linux can be supported with individual Toolfile that can be specified when running the install command.

# on macOS
ToolManager install --spec=Toolfile_macOS

# on Linux
ToolManager install --spec=Toolfile_Linux

Conclusion

The design described in this article powers the solution implemented at JET and has served our teams successfully since October 2023. JET has always preferred to implement in-house solutions where possible and sensible, and I can say that moving away from Homebrew was a blessing.

With this design, the work usually done by a package manager and a central spec repository is shifted to individual components that are only required to publish releases in zip archives, ideally via a release workflow.

By decentralising and requiring version pinning, we made ToolManager a simple yet powerful system for managing the installation of CLI tools.

It's very difficult to have GenAI not hallucinate when in comes to Swift Package Registry. No surprise there: the feature is definitely niche, has not been vastly adopted and there's a lack of examples online. As Dave put it, Swift Package Registries had an even rockier start compared to SPM.

I've recently implemented a Swift Package Registry on Artifactory for my team and I thought of summarising my experience here since it's still fresh in my head. While some details are left out, the happy path should be covered. I hope with this article to help you all indirectly by providing more material to the LLMs overlords.

Problem

The main problem that led us to look into Swift Package Registry is due to SPM deep-cloning entire Git repositories for each dependency, which became time-consuming.

Our CI jobs took a few minutes just to pull all the Swift packages. For dependencies with very large repositories, such as SendbirdUIKit (which is more than 2GB), one could rely on pre-compiled XCFrameworks as a workaround. Airbnb provides a workaround via the SPM-specific repo for Lottie.

A Swift Registry allows to serve dependencies as zip artifacts containing only the required revision, avoiding the deep clone of the git repositories.

What is a Swift Package Registry?

A Swift Package Registry is a server that stores and vends Swift packages by implementing SE-0292 and the corresponding specification. Instead of relying on Git repositories to source our dependencies, we can use a registry to download them as versioned archives (zip files).

swift-package-manager/Documentation/PackageRegistry/PackageRegistryUsage.md at main · swiftlang/swift-package-manager

The Package Manager for the Swift Programming Language - swiftlang/swift-package-manager

GitHubswiftlang

The primary advantages of using a Swift Package Registry are:

• Reduced CI/CD Pipeline Times: by fetching lightweight zip archives from the registry rather than cloning the entire repositories from GitHub.
• Improved Developer Machine Performance: the same time savings on CI are reflected on the developers' machines during dependency resolution.
• Availability: by hosting a registry, teams are no longer dependent on the availability of external source control systems like GitHub, but rather on internal ones (for example, self-hosted Artifactory).
• Security: injecting vulnerabilities in popular open-source projects is known as a supply chain attack and has become increasingly popular in recent years. A registry allows to adopt a process to trust the sources published on it.

Platforms

Apple has accepted the Swift Registry specification and implemented support to interact with registries within SPM but has left the implementation of actual registries to third-party platforms.

Apple is not in the business of providing a Swift Registry.

The main platform having adopted Swift Registries is Artifactory.

Artifactory, Your Swift Package Repository

JFrog now offers the first and only Swift binary package repository, enabling developers to use JFrog Artifactory for resolving Swift dependencies instead of enterprise source control (Git) systems.

JFroggiannit

although AWS CodeArtifact, Cloudsmith and Tuist provide support too:

New – Add Your Swift Packages to AWS CodeArtifact | Amazon Web Services

Starting today, Swift developers who write code for Apple platforms (iOS, iPadOS, macOS, tvOS, watchOS, or visionOS) or for Swift applications running on the server side can use AWS CodeArtifact to securely store and retrieve their package dependencies. CodeArtifact integrates with standard developer tools such as Xcode, xcodebuild, and the Swift Package Manager (the swift […]

Amazon Web ServicesSébastien Stormacq

Private, secure, hosted Swift registry

Cloudsmith offers secure, private Swift registries as a service, with cloud native performance. Book a demo today.

Cloudsmith

Announcing Tuist Registry

We’re thrilled to announce the launch of the Tuist Registry – a new feature that optimizes the resolution of Swift packages in your projects.

TuistMarek Fořt

The benefits are usually appealing to teams with large apps, hence it's reasonable to believe that only big companies have looked into adopting a registry successfully.

Artifactory Setup

Let's assume a JFrog Artifactory to host our Swift Package Registry exists at https://packages.acme.com. Artifactory support local, remote, and virtual repositories but a realistic setup consists of only local and virtual repositories.

Local Repositories are meant to be used for publishing dependencies from CI pipelines. Virtual Repositories are instead meant to be used for resolving (pulling) dependencies on both CI and the developers' machines. Remote repositories are not really relevant in a typical Swift Registry setup.

Following the documentation at https://jfrog.com/help/r/jfrog-artifactory-documentation/set-up-a-swift-registry, let's create 2 repositories with the following names:

• local repository: swift-local
• virtual repository: swift-virtual

Local Setup

To pull dependencies from the Swift Package Registry, we need to configure the local environment.

1. Set the Registry URL

First, we need to inform SPM about the existence of the registry. We can do this on a per-project basis or globally for the user account.

From a package's root directory, run the following command. This will create a .swiftpm/configuration/registries.json file within your project folder.

swift package-registry set "https://packages.acme.com/artifactory/api/swift/swift-virtual"

The resulting registries.json file will look like this:

{
  "authentication": {},
  "registries": {
    "[default]": {
      "supportsAvailability": false,
      "url": "https://packages.acme.com/artifactory/api/swift/swift-virtual"
    }
  },
  "version": 1
}

To set the registry for all your projects, use the --global flag.

swift package-registry set --global "https://packages.acme.com/artifactory/api/swift/swift-virtual"

This will create the configuration file at ~/.swiftpm/configuration/registries.json.

Xcode projects don't support project-level registries nor (in my experience) support scopes other than the default one (i.e. avoid using the --scope flag).

2. Authentication

To pull packages, authenticating with Artifactory is usually required. It's reasonable though that your company allows all artifacts from Artifactory to be read without authentication as long as one is connected to the company VPN.

In cases where authentication is required, SPM uses a .netrc file in the home directory to find credentials for remote servers. This file is a standard way to handle login information for various network protocols.

Using a token generated from the Artifactory dashboard, the line to add to the .netrc file would be:

machine packages.acme.com login <your_artifactory_username> password <your_artifactory_token>

Alternatively, it's possible to log in using the swift package-registry login command. This command securely stores your token in the system's keychain.

swift package-registry login "https://packages.acme.com/artifactory/api/swift/swift-virtual" \
  --token <token>

# or

swift package-registry login "https://packages.acme.com/artifactory/api/swift/swift-virtual" \
  --username <username> \
  --password <token_treated_as_password>

CI/CD Setup

On CI, the setup is slightly different as the goals are:

• to resolve dependencies in CI/CD jobs
• to publish new package versions in CD jobs for both internal and external dependencies

The steps described for the local setup are valid for the resolution on CI too. The interesting part here is how publishing is done. I will assume the usage of GitHub Actions.

1. Retrieving the Artifactory Token

The JFrog CLI can be used via the setup-jfrog-cli action to authenticate using the most appropriate method. You might want to wrap the action in a custom composable one exporting the token as the output of a step:

TOKEN=$(jf config export) 
echo "::add-mask::$TOKEN"
echo "artifactory-token=$TOKEN">> "$GITHUB_OUTPUT"

2. Logging into the Registry

The CI job must log in to the local repository (swift-local) to gain push permissions. The token retrieved in the previous step is used for this purpose.

swift package-registry login \
  "https://packages.acme.com/artifactory/api/swift/swift-local" \
  --token ${{ steps.get-token.outputs.artifactory-token }}

3. Publishing Packages

Swift Registry requires archives created with the swift package archive-source command from the dependency folder. E.g.

swift package archive-source -o "Alamofire-5.10.2.zip"

We could avoid creating the archive and instead download it directly from GitHub releases.

curl -L -o Alamofire-5.10.1.zip \
  https://github.com/Alamofire/Alamofire/archive/refs/tags/5.10.1.zip

Uploading the archive can then be done by using the JFrog CLI that needs customization via the setup-jfrog-cli action. If going down this route, the upload command would be:

jf rt upload Alamofire-5.10.1.zip \
  https://packages.acme.com/artifactory/api/swift/swift-local/acme/Alamofire/Alamofire-5.10.1.zip

There is a specific structure to respect:

<REPOSITORY>/<SCOPE>/<NAME>/<NAME>-<VERSION>.zip

which is the last part of the above URL:

swift-local/acme/Alamofire/Alamofire-5.10.1.zip

Too bad that using the steps above causes a downstream problem with SPM not being able to resolve the dependencies in the registry. I tried extensively and couldn't find the reason why SPM wouldn't be happy with how the packages were published. I might have missed something but eventually I necessarily had to switch to use the publish command.

Using the swift package-registry publish command instead, doesn't present this issue hence it's the solution adopted in this workflow.

swift package-registry publish acme.Alamofire 5.10.1 \
  --url https://packages.acme.com/artifactory/api/swift/swift-local \
  --scratch-directory $(mktemp -d)

To verify the upload and indexing succeeded, check that the uploaded *.zip artifact is available and that the .swift exists (indication that the indexing has occurred). If the specific structure is not respected, the .swift folder wouldn't be generated.

Consuming Packages from the Registry

Packages

The easiest and only documented way to consume a package from a registry is via a Package. In the Package.swift file, declare dependencies using the .package(id:from:) syntax to declare a registry-based dependency. The id is a combination of the scope and the package name.

    ...
    dependencies: [
        .package(id: "acme.Alamofire", from: "5.10.1"),
    ],
    targets: [
        .target(
            name: "MyApp",
            dependencies: [
                .product(name: "Alamofire", package: "acme.Alamofire"),
            ]
        ),
        ...
    ]
)

Run swift package resolve or simply build the Package in Xcode to pull the dependencies.

You might bump into transitive dependencies (i.e. dependencies listed in the Package.swift files of the packages published on the registry) pointing to GitHub. In this case, it'd be great to instruct SPM to use the corresponding versions on the registry. The --replace-scm-with-registry flag is designed to work for the entire dependency graph, including transitive dependencies.

The cornerstone of associating a registry-hosted package with its GitHub origin is the package-metadata.json file. This file allows to provide essential metadata about the packages at the time of publishing (the --metadata-path flag of the publish command defaults to package-metadata.json).

Crucially, it includes a field to specify the source control repository URLs. When swift package resolve --replace-scm-with-registry is executed, SPM queries the configured registry. The registry then uses the information from the package-metadata.json to map the package identity to its corresponding GitHub URL, enabling a smooth and transparent resolution process.

The metadata file must conform to the JSON schema defined in SE-0391. It is recommended to include all URL variations (e.g., SSH, HTTPS) for the same repository. E.g.

{
  "repositoryURLs": [
    "https://github.com/Alamofire/Alamofire",
    "https://github.com/Alamofire/Alamofire.git",
    "git@github.com:Alamofire/Alamofire.git"
  ]
}

Printing the dependencies should confirm the source of the dependencies:

swift package show-dependencies --replace-scm-with-registry

When loading a package with Xcode, the flag can be enabled via an environment variable in the scheme

IDEPackageDependencySCMToRegistryTransformation=useRegistryIdentityAndSources

Too bad that for packages, the schemes won't load until SPM completes the resolution hence running the following from the terminal would address the issue:

defaults write com.apple.dt.Xcode IDEPackageDependencySCMToRegistryTransformation useRegistryIdentityAndSources

that can be unset with:

defaults delete com.apple.dt.Xcode IDEPackageDependencySCMToRegistryTransformation

Xcode

It's likely that you'll want to use the registry from Xcode projects for direct dependencies. If using the Tuist registry, it seems you would be able to leverage a Package Collection to add dependencies from the registry from the Xcode UI. Note that until Xcode 26 Beta 1, it's not possible to add registry dependencies directly in the Xcode UI, but if you use Tuist to generate your project (as you should), you can use the Package.registry (introduced with https://github.com/tuist/tuist/pull/7225). E.g.

let project = Project(
    ...
    packages: [
        .registry(
            identifier: "acme.Alamofire",
            requirement: .exact(Version(stringLiteral: "5.10.1"))
        )
    ],
    ...
)

If not using Tuist, you'd have to rely on setting IDEPackageDependencySCMToRegistryTransformation either as an environment variable in the scheme or globally via the terminal.

You can also use xcodebuild to resolve dependencies using the correct flag:

xcodebuild \
  -resolvePackageDependencies \
  -packageDependencySCMToRegistryTransformation useRegistryIdentityAndSources

Conclusions

We’ve found that using an in-house Swift registry drastically reduces dependency resolution time and size on disk by downloading only the required revision instead of the entire, potentially large, Git repository. This improvement benefits both CI pipelines and developers’ local environments. Additionally, registries help mitigate the risk of supply chain attacks.

As of this writing, Swift registries are not widely adopted, which is reflected in the limited number of platforms that support them. It also shows various bugs I myself bumped into when using particular configurations.

It's unclear whether adoption will grow and uncertain if Apple will ever address the issues reported by the community, but when a functioning setup is put in place, registries offer an efficient and secure alternative to using XCFrameworks in production builds and reduce both memory and time footprints.

Originally published on the Just Eat Takeaway Engineering Blog.

How Just Eat Takeaway.com leverage AWS, Packer, Terraform and GitHub Actions to manage a CI stack of macOS runners.

Problem

At Just Eat Takeaway.com (JET), our journey through continuous integration (CI) reflects a landscape of innovation and adaptation. Historically, JET’s multiple iOS teams operated independently, each employing their distinct CI solutions.

The original Just Eat iOS and Android teams had pioneered an in-house CI solution anchored in Jenkins. This setup, detailed in our 2021 article, served as the backbone of our CI practices up until 2020. It was during this period that the iOS team initiated a pivotal migration: moving from in-house Mac Pros and Mac Minis to AWS EC2 macOS instances.

Fast forward to 2023, a significant transition occurred within our Continuous Delivery Engineering (CDE) Platform Engineering team. The decision to adopt GitHub Actions company-wide has marked the end of our reliance on Jenkins while other teams are in the process of migrating away from solutions such as CircleCI and GitLab CI. This transition represented a fundamental shift in our CI philosophy. By moving away from Jenkins, we eliminated the need to maintain an instance for the Jenkins server and the complexities of managing how agents connected to it. Our focus then shifted to transforming our Jenkins pipelines into GitHub Actions workflows.

This transformation extended beyond mere tool adoption. Our primary goal was to ensure that our macOS instances were not only scalable but also configured in code. We therefore enhanced our global CI practices and set standards across the entire company.

Desired state of CI

As we embarked on our journey to refine and elevate our CI process, we envisioned a state-of-the-art CI system. Our goals were ambitious yet clear, focusing on scalability, automation, and efficiency. At the time of implementing the system, no other player in the industry seemed to have implemented the complete solution we envisioned.

Below is a summary of our desired CI state:

• Instance setup in code: One primary objective was to enable the definition of the setup of the instances entirely in code. This includes specifying macOS version, Xcode version, Ruby version, and other crucial configurations. For this purpose, the HashiCorp tool Packer, emerged once again as an ideal solution, offering the flexibility and precision we required.
• IaC (Infrastructure as Code) for macOS instances: To define the infrastructure of our fleet of macOS instances, we leaned towards Terraform, another HashiCorp tool. Terraform provided us with the capability to not only deploy but also to scale and migrate our infrastructure seamlessly, crucially maintaining its state.
• Auto and Manual Scaling: We wanted the ability to dynamically create CI runners based on demand, ensuring that resources were optimally utilized and available when needed. To optimize resource utilization, especially during off-peak hours, we desired an autoscaling feature. Scaling down our CI runners on weekends when developer activity is minimal was critical to be cost-effective.
• Automated Connection to GitHub Actions: We aimed for the instances to automatically connect to GitHub Actions as runners upon deployment. This automation was crucial in eliminating manual interventions via SSH or VNC.
• Multi-Team Use: Our vision included CI runners that could be easily used by multiple teams across different time zones. This would not only maximize the utility of our infrastructure but also encourage reuse and standardization.
• Centralized Management via GitHub Actions: To further streamline our CI processes, we intended to run all tasks through GitHub Actions workflows. This approach would allow the teams to self-serve and alleviate the need for developers to use Docker or maintain local environments.

Getting to the desired state was a journey that presented multiple challenges and constant adjustments to make sure we could migrate smoothly to a new system.

Instance setup in code

We implemented the desired configuration with Packer leveraging a number of Shell Provisioners and variables to configure the instance. Here are some of the configuration steps:

• Set user password (to allow remote desktop access)
• Resize the partition to use all the space available on the EBS volume
• Start the Apple Remote Desktop agent and enable remote desktop access
• Update Brew & Install Brew packages
• Install CloudWatch agent
• Install rbenv/Ruby/bundler
• Install Xcode versions
• Install GitHub Actions actions-runner
• Copy scripts to connect to GitHub Actions as a runner
• Copy daemon to start the GitHub Actions self-hosted runner as a service
• Set macos-init modules to perform provisioning of the first launch

While the steps above are naturally configuration steps to perform when creating the AMI, the macos-init modules include steps to perform once the instance becomes available.

The create_ami workflow accepts inputs that are eventually passed to Packer to generate the AMI.

packer build \
  --var ami_name_prefix=${{ env.AMI_NAME_PREFIX }} \
  --var region=${{ env.REGION }} \
  --var subnet_id=${{ env.SUBNET_ID }} \
  --var vpc_id=${{ env.VPC_ID }} \
  --var root_volume_size_gb=${{ env.ROOT_VOLUME_SIZE_GB }} \
  --var macos_version=${{ inputs.macos-version}} \
  --var ruby_version=${{ inputs.ruby-version }} \
  --var xcode_versions='${{ steps.parse-xcode-versions.outputs.list }}' \
  --var gha_version=${{ inputs.gha-version}} \
  bare-metal-runner.pkr.hcl

Different teams often use different versions of software, like Xcode. To accommodate this, we permit multiple versions to be installed on the same instance. The choice of which version to use is then determined within the GitHub Actions workflows.

The seamless generation of AMIs has proven to be a significant enabler. For example, when Xcode 15.1 was released, we executed this workflow the same evening. In just over two hours, we had an AMI ready to deploy all the runners (it usually takes 70–100 minutes for a macOS AMI with 400GB of EBS volume to become ready after creation). This efficiency enabled our teams to use the new Xcode version just a few hours after its release.

IaC (Infrastructure as Code) for macOS instances

Initially, we used distinct Terraform modules for each instance to facilitate the deployment and decommissioning of each one. Given the high cost of EC2 Mac instances, we managed this process with caution, carefully balancing host usage while also being mindful of the 24-hour minimum allocation time.

We ultimately ended up using Terraform to define a single infrastructure (i.e. a single Terraform module) defining resources such as:

• aws_key_pair, aws_instance, aws_ami
• aws_security_group, aws_security_group_rule
• aws_secretsmanager_secret
• aws_vpc, aws_subnet
• aws_cloudwatch_metric_alarm
• aws_sns_topic, aws_sns_topic_subscription
• aws_iam_role, aws_iam_policy, aws_iam_role_policy_attachment, aws_iam_instance_profile

A crucial part was to use count in aws_instance, setting the value of a variable passed in from deploy_infra workflow. Terraform performs the necessary scaling upon changing the value.

We have implemented a workflow to perform Terraform apply and destroy commands for the infrastructure. Only the AMI and the number of instances are required as inputs.

terraform ${{ inputs.command }} \
  --var ami_name=${{ inputs.ami-name }} \
  --var fleet_size=${{ inputs.fleet-size }} \
  --auto-approve

Using the name of the AMI instead of the ID allows us to use the most recent one that was generated, useful in case of name clashes.

variable "ami_name" {
  type = string
}

variable "fleet_size" {
  type = number
}

data "aws_ami" "bare_metal_gha_runner" {
  most_recent = true

  filter {
    name   = "name"
    values = ["${var.ami_name}"]
  }
  
  ...
}

resource "aws_instance" "bare_metal" {
  count         = var.fleet_size
  ami           = data.aws_ami.bare_metal_gha_runner.id
  instance_type = "mac2.metal"
  tenancy       = "host"
  key_name      = aws_key_pair.bare_metal.key_name
  ...
}

Instead of maintaining multiple CI instances with varying software configurations, we concluded that it’s simpler and more efficient to have a single, standardised setup. While teams still have the option to create and deploy their unique setups, a smaller, unified system allows for easier support by a single global configuration.

Auto and Manual Scaling

The deploy_infra workflow allows us to scale on demand but it doesn’t release the underlying dedicated hosts which are the resources that are ultimately billed.

The autoscaling solution provided by AWS is great for VMs but gets sensibly more complex when actioned on dedicated hosts. Auto Scaling groups on macOS instances would require a Custom Managed License, a Host Resource Group and, of course, a Launch Template. Using only AWS services appears to be a lot of work to pull things together and the result wouldn’t allow for automatic release of the dedicated hosts.

AirBnb mention in their Flexible Continuous Integration for iOS article that an internal scaling service was implemented:

An internal scaling service manages the desired capacity of each environment’s Auto Scaling group.

Some articles explain how to set up Auto Scaling groups for mac instances (see 1 and 2) but after careful consideration, we agreed that implementing a simple scaling service via GitHub Actions (GHA) was the easiest and most maintainable solution.

We implemented 2 GHA workflows to fully automate the weekend autoscaling:

• Upscaling workflow to n, triggered at a specific time at the beginning of the working week
• Downscaling workflow to 1, triggered at a specific time at the beginning of the weekend

We retain the capability for manual scaling, which is essential for situations where we need to scale down, such as on bank holidays, or scale up, like on release cut days, when activity typically exceeds the usual levels.

Additionally, we have implemented a workflow that runs multiple times a day and tries to release all available hosts without an instance attached. This step lifts us from the burden of having to remember to release the hosts. Dedicated hosts take up to 110 minutes to move from the Pending to the Available state due to the scrubbing workflow performed by AWS.

Manual scaling can be executed between the times the autoscaling workflows are triggered and they must be resilient to unexpected statuses of the infrastructure, which thankfully Terraform takes care of.

Both down and upscaling are covered in the following flowchart:

The autoscaling values are defined as configuration variables in the repo settings:

It usually takes ~8 minutes for an EC2 mac2.metal instance to become reachable after creation, meaning that we can redeploy the entire infrastructure very quickly.

Automated Connection to GitHub Actions

We provide some user data when deploying the instances.

resource "aws_instance" "bare_metal" {
  ami       = data.aws_ami.bare_metal_gha_runner.id
  count     = var.fleet_size
  ...
  user_data = <<EOF
{
    "github_enterprise": "<GHE_ENTERPRISE_NAME>",
    "github_pat_secret_manager_arn": ${data.aws_secretsmanager_secret_version.ghe_pat.arn},
    "github_url": "<GHE_ENTERPRISE_URL>",
    "runner_group": "CI-MobileTeams",
    "runner_name": "bare-metal-runner-${count.index + 1}"
}
  EOF

The user data is stored in a specific folder by macos-init and we implement a module to copy the content to ~/actions-runner-config.json.

### Group 10 ###
[[Module]]
    Name = "Create actions-runner-config.json from userdata"
    PriorityGroup = 10
    RunPerInstance = true
    FatalOnError = false
    [Module.Command]
        Cmd = ["/bin/zsh", "-c", 'instanceId="$(curl http://169.254.169.254/latest/meta-data/instance-id)"; if [[ ! -z $instanceId ]]; then cp /usr/local/aws/ec2-macos-init/instances/$instanceId/userdata ~/actions-runner-config.json; fi']
        RunAsUser = "ec2-user"

which is in turn used by the configure_runner.sh script to configure the GitHub Actions runner.

#!/bin/bash

GITHUB_ENTERPRISE=$(cat $HOME/actions-runner-config.json | jq -r .github_enterprise)
GITHUB_PAT_SECRET_MANAGER_ARN=$(cat $HOME/actions-runner-config.json | jq -r .github_pat_secret_manager_arn)
GITHUB_PAT=$(aws secretsmanager get-secret-value --secret-id $GITHUB_PAT_SECRET_MANAGER_ARN | jq -r .SecretString)
GITHUB_URL=$(cat $HOME/actions-runner-config.json | jq -r .github_url)
RUNNER_GROUP=$(cat $HOME/actions-runner-config.json | jq -r .runner_group)
RUNNER_NAME=$(cat $HOME/actions-runner-config.json | jq -r .runner_name)

RUNNER_JOIN_TOKEN=` curl -L \
  -X POST \
  -H "Accept: application/vnd.github+json" \
  -H "Authorization: Bearer $GITHUB_PAT"\
  $GITHUB_URL/api/v3/enterprises/$GITHUB_ENTERPRISE/actions/runners/registration-token | jq -r '.token'`

MACOS_VERSION=`sw_vers -productVersion`

XCODE_VERSIONS=`find /Applications -type d -name "Xcode-*" -maxdepth 1 \
  -exec basename {} \; \
  | tr '\n' ',' \
  | sed 's/,$/\n/' \
  | sed 's/.app//g'`

$HOME/actions-runner/config.sh \
  --unattended \
  --url $GITHUB_URL/enterprises/$GITHUB_ENTERPRISE \
  --token $RUNNER_JOIN_TOKEN \
  --runnergroup $RUNNER_GROUP \
  --labels ec2,bare-metal,$RUNNER_NAME,macOS-$MACOS_VERSION,$XCODE_VERSIONS \
  --name $RUNNER_NAME \
  --replace

The above script is run by a macos-init module.

### Group 11 ###
[[Module]]
    Name = "Configure the GHA runner"
    PriorityGroup = 11
    RunPerInstance = true
    FatalOnError = false
    [Module.Command]
        Cmd = ["/bin/zsh", "-c", "/Users/ec2-user/configure_runner.sh"]
        RunAsUser = "ec2-user"

The GitHub documentation states that it’s possible to create a customized service starting from a provided template. It took some research and various attempts to find the right configuration that allows the connection without having to log in in the UI (over VNC) which would represent a blocker for a complete automation of the deployment. We believe that the single person who managed to get this right is Sébastien Stormacq who provided the correct solution.

The connection to GHA can be completed with 2 more modules that install the runner as a service and load the custom daemon.

### Group 12 ###
[[Module]]
    Name = "Run the self-hosted runner application as a service"
    PriorityGroup = 12
    RunPerInstance = true
    FatalOnError = false
    [Module.Command]
        Cmd = ["/bin/zsh", "-c", "cd /Users/ec2-user/actions-runner && ./svc.sh install"]
        RunAsUser = "ec2-user"

### Group 13 ###
[[Module]]
    Name = "Launch actions runner daemon"
    PriorityGroup = 13
    RunPerInstance = true
    FatalOnError = false
    [Module.Command]
        Cmd = ["sudo", "/bin/launchctl", "load", "/Library/LaunchDaemons/com.justeattakeaway.actions-runner-service.plist"]
        RunAsUser = "ec2-user"

Using a daemon instead of an agent (see Creating Launch Daemons and Agents), doesn’t require us to set up any auto-login which on macOS is a bit of a tricky procedure and is best avoided also for security reasons. The following is the content of the daemon for completeness.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
  <dict>
    <key>Label</key>
    <string>com.justeattakeaway.actions-runner-service</string>
    <key>ProgramArguments</key>
    <array>
      <string>/Users/ec2-user/actions-runner/runsvc.sh</string>
    </array>
    <key>UserName</key>
    <string>ec2-user</string>
    <key>GroupName</key>
    <string>staff</string>
    <key>WorkingDirectory</key>
    <string>/Users/ec2-user/actions-runner</string>
    <key>RunAtLoad</key>
    <true/>    
    <key>StandardOutPath</key>
    <string>/Users/ec2-user/Library/Logs/com.justeattakeaway.actions-runner-service/stdout.log</string>
    <key>StandardErrorPath</key>
    <string>/Users/ec2-user/Library/Logs/com.justeattakeaway.actions-runner-service/stderr.log</string>
    <key>EnvironmentVariables</key>
    <dict> 
      <key>ACTIONS_RUNNER_SVC</key>
      <string>1</string>
    </dict>
    <key>ProcessType</key>
    <string>Interactive</string>
    <key>SessionCreate</key>
    <true/>
  </dict>
</plist>

Not long after the deployment, all the steps above are executed and we can appreciate the runners appearing as connected.

Multi-Team Use

We start the downscaling at 11:59 PM on Fridays and start the upscaling at 6:00 AM on Mondays. These times have been chosen in a way that guarantees a level of service to teams in the UK, the Netherlands (GMT+1) and Canada (Winnipeg is on GMT-6) accounting for BST (British Summer Time) and DST (Daylight Saving Time) too. Times are defined in UTC in the GHA workflow triggers and the local time of the runner is not taken into account.

Since the instances are used to build multiple projects and tools owned by different teams, one problem we faced was that instances could get compromised if workflows included unsafe steps (e.g. modifications to global configurations).

GitHub Actions has a documentation page about Hardening self-hosted runners specifically stating:

Self-hosted runners for GitHub do not have guarantees around running in ephemeral clean virtual machines, and can be persistently compromised by untrusted code in a workflow.

We try to combat such potential problems by educating people on how to craft workflows and rely on the quick redeployment of the stack should the instances break.

We also run scripts before and after each job to ensure that instances can be reused as much as possible. This includes actions like deleting the simulators’ content, derived data, caches and archives.

Centralized Management via GitHub Actions

The macOS runners stack is defined in a dedicated macOS-runners repository. We implemented GHA workflows to cover the use cases that allow teams to self-serve:

• create macOS AMI
• deploy CI
• downscale for the weekend*
• upscale for the working week*
• release unused hosts*

* run without inputs and on a scheduled trigger

The runners running the jobs in this repo are small t2.micro Linux instances and come with the AWSCLI installed. An IAM instance role with the correct policies is used to make sure that aws ec2 commands allocate-hosts, describe-hosts and release-hosts could execute and we used jq to parse the API responses.

A note on VM runners

In this article, we discussed how we’ve used bare metal instances as runners. We have spent a considerable amount of time investigating how we could leverage the Virtualization framework provided by Apple to create virtual machines via Tart.

If you’ve grasped the complexity of crafting a CI system of runners on bare metal instances, you can understand that introducing VMs makes the setup sensibly more convoluted which would be best discussed in a separate article.

While a setup with Tart VMs has been implemented, we realised that it’s not performant enough to be put to use. Using VMs, the number of runners would double but we preferred to have native performance as the slowdown is over 40% compared to bare metal. Moreover, when it comes to running heavy UI test suites like ours, tests became too flaky.

Testing the VMs, we also realised that the standard values of Throughput and IOPS on the EBS volume didn’t seem to be enough and caused disk congestion resulting in an unacceptable slowdown in performance.

Here is a quick summary of the setup and the challenges we have faced.

• Virtual runners require 2 images: one for the VMs (tart) and one for the host (AMI).
• We use Packer to create VM images (Vanilla, Base, IDE, Tools) with the software required based on the templates provided by Tart and we store the OCI-compliant images on ECR. We create these images on CI with dedicated workflows similar to the one described earlier for bare metal but, in this case, macOS runners (instead of Linux) are required as publishing to ECR is done with tart which runs on macOS. Extra policies are required on the instance role to allow the runner to push to ECR (using temporary_iam_instance_profile_policy_document in Packer’s Amazon EBS).
• Apple set a limit to the number of VMs that can be run on an instance to 2, which would allow to double the size of the fleet of runners. Creating AMIs hosting 2 VMs is done with Packer and steps include cloning the image from ECR and configuring macos-init modules to run daemons to run the VMs via Tart.
• Deploying a virtual CI infrastructure is identical to what has already been described for bare metal.
• Connecting to and interfacing with the VMs happens from within the host. Opening SSH and especially VNC sessions from within the bare metal instances can be very confusing.
• The version of macOS on the host and the one on the VMs could differ. The version used on the host must be provided with an AMI by AWS, while the version used on the VMs is provided by Apple in IPSW files (see ipsw.me).
• The GHA runners run on the VMs meaning that the host won’t require Xcode installed nor any other software used by the workflows.
• VMs don’t allow for provisioning meaning we have to share configurations with the VMs via shared folders on the host with the — dir flag which causes extra setup complexity.
• VMs can’t easily run the GHA runner as a service. The svc script requires the runner to be configured first, an operation that cannot be done during the provisioning of the host. We therefore need to implement an agent ourselves to configure and connect the runner in a single script.
• To have UI access (a-la VNC) to the VMs, it’s first required to stop the VMs and then run them without the --no-graphics flag. At the time of writing, copy-pasting won’t work even if using the --vnc or --vnc-experimental flags.
• Tartelet is a macOS app on top of Tart that allows to manage multiple GitHub Actions runners in ephemeral environments on a single host machine. We didn’t consider it to avoid relying on too much third-party software and because it doesn’t have yet GitHub Enterprise support.
• Worth noting that the Tart team worked on an orchestration solution named Orchard that seems to be in its initial stage.

Conclusion

In 2023 we have revamped and globalised our approach to CI. We have migrated from Jenkins to GitHub Actions as the CI/CD solution of choice for the whole group and have profoundly optimised and improved our pipelines introducing a greater level of job parallelisation.

We have implemented a brand new scalable setup for bare metal macOS runners leveraging the HashiCorp tools Packer and Terraform. We have also implemented a setup based on Tart virtual machines.

We have increased the size of our iOS team over the past few years, now including more than 40 developers, and still managed to be successful with only 5 bare metal instances on average, which is a clear statement of how performant and optimised our setup is.

We have extended the capabilities of our Internal Developer Platform with a globalised approach to provide macOS runners; we feel that this setup will stand the test of time and serve well various teams across JET for years to come.

Fastlane is widely used by iOS teams all around the world. It became the standard de facto to automate common tasks such as building apps, running tests, and uploading builds to App Store Connect. Fastlane has been recently moved under the Mobile Native Foundation which is amazing as Google wasn't actively maintaining the project.

At Just Eat Takeaway, we have implemented an extensive number of custom lanes to perform domain-specific tasks and used them from our CI.

The major problem with Fastlane is that it's written in Ruby. When it was born, using Ruby was a sound choice but iOS developers are not necessarily familiar with such language which represents a barrier to contributing and writing lanes.

While Fastlane.swift, a version of Fastlane in Swift, has been in beta for years, it's not a rewrite in Swift but rather a "solution on top" meaning that developers and CI systems still have to rely on Ruby, install related software (rbenv or rvm) and most likely maintain a Gemfile. The average iOS dev knows well that Ruby environments are a pain to deal with and have caused an infinite number of headaches.

In recent years, Apple has introduced technologies that would enable a replacement of Fastlane using Swift:

• Swift Package Manager (SPM)
• Swift Argument Parser (SAP)

Being myself a big fan of CLI tools written in Swift, I soon started maturing the idea of a Fastlane rewrite in Swift in early 2022. I circulated the idea with friends and colleagues for months and the sentiment was clear: it's time for a fresh simil-Fastlane tool written in Swift.

Journey

Towards the end of 2022, I was determined to start this project. I teamed up with 2 iOS devs (not working at Just Eat Takeaway) and we started working on a design. I was keen on calling this project "Swiftlane" but the preference seemed to be for the name "Interstellar" which was eventually shortened into "Stellar".

Fastlane has the concept of Actions and I instinctively thought that in Swift-land, they could take the form of SPM packages. This would make Stellar a modular system with pluggable components.

For example, consider the Scan action in Fastlane. It could be a package that solely solves the same problem around testing. My goal was not to implement the plethora of existing Fastlane actions but rather to create a system that allows plugging in any package building on macOS. A sound design of such system was crucial.

The Stellar ecosystem I had in mind was composed of 4 parts:

Actions

Actions are the basic building blocks of the ecosystem. They are packages that define a library product. An action can do anything, from taking care of build tasks to integrating with GitHub.

Actions are independent packages that have no knowledge of the Stellar system, which treats them as pluggable components to create higher abstractions.

Ideally, actions should expose an executable product (the CLI tool) using SAP calling into the action code. This is not required by Stellar but it’s advisable as a best practice.

Official Actions would be hosted in the Stellar organisation on GitHub. Custom Actions could be created using Stellar.

Tasks

Tasks are specific to a project and implemented by the project developers. They are SAP ParsableCommand or AsyncParsableCommand which use actions to construct complex logic specific to the needs of the project.

Executor

Executor is a command line tool in the form of a package generated by Stellar. It’s the entry point to the user-defined tasks. Invoking tasks on the Executor is like invoking lanes in Fastlane.

Both developers and CI would interface with the Executor (masked as Stellar) to perform all operations. E.g.

stellar setup_environment --developer-mode
stellar run_unit_tests module=OrderHistory
stellar setup_demo_app module=OrderHistory
stellar run_ui_tests module=OrderHistory device="iPhone 15 Pro"

Stellar CLI

Stellar CLI is a command line tool that takes care of the heavy lifting of dealing with the Executor and the Tasks. It allows the integration of Stellar in a project and it should expose the following main commands:

• init: initialise the project by creating an Exectutor package in the .stellar folder
• build: builds the Executor generating a binary that is shared with the team members and used by CI
• create-action: scaffolding to create a new action in the form of a package
• create-task: scaffolding to create a new task in the form of a package
• edit: opens the Executor package for editing, similar to tuist edit

This design was presented to a restricted group of devs at Just Eat Takeaway and it didn't take long to get an agreement on it. It was clear that once Stellar was completed, we would have integrated it in the codebase.

Wider design

I believe that a combination of CLI tools can create complex, templateable and customizable stacks to support the creation and growth of iOS codebases.

Based on the experience developed at JET working on a large modular project with lots of packages, helper tools and optimised CI pipelines, I wanted Stellar to be eventually part of a set of tools taking the name “Stellar Tools” that could enable the creation and the management of large codebases.

Something like the following:

• Tuist: generates projects and workspaces programmatically
• PackageGenerator: generates packages using a DSL
• Stacker: creates a modular iOS project based on a DSL
• Stellar: automate tasks
• Workflows: generates GitHub Actions workflows that use Stellar

From my old notes:

Current state

After a few months of development within this team (made of devs not working at Just Eat Takeaway), I realised things were not moving in the direction I desired and I decided it was not beneficial to continue the collaboration with the team. We stopped working on Stellar mainly due to different levels of commitment from each of us and focus on the wrong tasks signalling a lack of project management from my end. For example, a considerable amount of time and effort went into the implementation of a version management system (vastly inspired by the one used in Tuist) that was not part of the scope of the Stellar project.

The experience left me bitter and demotivated, learning that sometimes projects are best started alone. We made the repo public on GitHub aware that it was far from being production-ready but in my opinion, it's no doubt a nice, inspiring, MVP.

GitHub - StellarTools/Stellar

Contribute to StellarTools/Stellar development by creating an account on GitHub.

GitHubStellarTools

GitHub - StellarTools/ActionDSL

Contribute to StellarTools/ActionDSL development by creating an account on GitHub.

GitHubStellarTools

The intent was then to progress on my own or with my colleagues at JET. As things evolved in 2023, we embarked on big projects that continued to evolve the platform such as a massive migration to GitHub Actions. To this day, we still plan to remove Fastlane as our vision is to rely on external dependencies as little as possible but there is no plan to use Stellar as-is. I suspect that, for the infrastructure team at JET, things will evolve in a way that sees more CLI tools being implemented and more GitHub actions using them.

Amazon Web Services (AWS) provides EC2 Mac instances commonly used as CI workers. Configuring them can be either a manual or an automated process, depending on the DevOps and Platform Engineering experience in your company. No matter what process you adopt, it is sometimes useful to log into the instances to investigate problems.

EC2macConnector is a CLI tool written in Swift that simplifies the process of connecting over SSH and VNC for DevOps engineers, removing the need of updating private keys and maintaining the list of IPs that change across deployment cycles.

Connecting to EC2 Mac instances without EC2macConnector

AWS documentation describes the steps needed to allow connecting via VNC:

1. Start the Apple Remote Desktop agent and enable remote desktop access on the instance
2. Set the password for the ec2-user user on the instance to allow connecting over VNC
3. Start an SSH session
4. Connect over VNC

Assuming steps 1 and 2 and done, steps 3 and 4 are usually manual and repetitive: the private keys and IPs usually change across deployments which could happen frequently, even daily.

Here is how to start an SSH session in the terminal binding a port locally:

ssh ec2-user@<instance_IP> \
  -L <local_port>:localhost:5900 \
  -i <path_to_privae_key> \

To connect over VNC you can type the following in Finder → Go → Connect to Server (⌘ + K) and click Connect:

vnc://ec2-user@localhost:<local_port>

or you could create a .vncloc file with the following content and simply open it:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "<http://www.apple.com/DTDs/PropertyList-1.0.dtd>">
<plist version="1.0">
<dict>
	<key>URL</key>
	<string>vnc://ec2-user@localhost:<local_port></string>
</dict>
</plist>

If you are a system administrator, you might consider EC2 Instance Connect, but sadly, in my experience, it's not a working option for EC2 Mac instances even though I couldn't find evidence confirming or denying this statement.

Administrators could also consider using Apple Remote Desktop which comes with a price tag of $/£79.99.

Connecting to EC2 Mac instances with EC2macConnector

EC2macConnector is a simple and free tool that works in 2 steps:

• the configure command fetches the private keys and the IP addresses of the running EC2 Mac instances in a given region, and creates files using the said information to connect over SSH and VNC:

ec2macConnector configure \
  --region <aws_region> \
  --secrets-prefix <mac_metal_private_keys_prefix>

Read below or the README for more information on the secrets prefix value.

• the connect command connects to the instances via SSH or VNC.

ec2macConnector connect --region <aws_region> <fleet_index>

ec2macConnector connect --region <aws_region> <fleet_index> --vnc

As with any tool written using ArgumentParser, use the --help flag to get more information.

Requirements

There are some requirements to respect for the tool to work:

Permissions

EC2macConnector requires AWS credentials either set as environment variables (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY) or configured in ~/.aws/credentials via the AWS CLI. Environment variables take precedence.

The user must be granted the following permissions:

• ec2:DescribeInstances
• secretsmanager:ListSecrets
• secretsmanager:GetSecretValue

EC2 instances

The EC2 Mac instances must have the EC2macConnector:FleetIndex tag set to the index of the instance in the fleet. Indexes should start at 1. Instances that don't have the said tag will be ignored.

Secrets and key pairs formats

EC2macConnector assumes that the private key for each instance key pair is stored in SecretsManagers. The name of the key pair could and should be different from the secret ID. For example, the instance key pair should include an incremental number also part of the corresponding secret ID.

Consider that the number of Mac instances in an AWS account is limited and it's convenient to refer to them using an index starting at 1. It's good practice for the secret ID to also include a nonce as secrets with the same name cannot be recreated before the deletion period has elapsed, allowing frequent provisioning-decommissioning cycles.

For the above reasons, EC2macConnector assumes the following formats are used:

• instance key pairs: <keypair_prefix>_<index_of_instance_in_fleet> e.g. mac_instance_key_pair_5
• secret IDs: <secrets_prefix>_<index_of_instance_in_fleet>_<nonce> e.g. private_key_mac_metal_5_dx9Wna73B

EC2macConnector Under the hood

The configure command:

• downloads the private keys in the ~/.ssh folder
• creates scripts to connect over SSH in ~/.ec2macConnector/<region>/scripts
• creates vncloc files to connect over VNC in ~/.ec2macConnector/<region>/vnclocs

➜  .ec2macConnector tree ~/.ssh
/Users/alberto/.ssh
├── mac_metal_1_i-08e4ffd8e9xxxxxxx
├── mac_metal_2_i-07bfff1f52xxxxxxx
├── mac_metal_3_i-020d680610xxxxxxx
├── mac_metal_4_i-08516ac980xxxxxxx
├── mac_metal_5_i-032bedaabexxxxxxx
├── config
├── known_hosts
└── ...

The connect command:

• runs the scripts (opens new shells in Terminal and connects to the instances over SSH)
• opens the vncloc files

➜  .ec2macConnector tree ~/.ec2macConnector
/Users/alberto/.ec2macConnector
└── us-east-1
    ├── scripts
    │   ├── connect_1.sh
    │   ├── connect_2.sh
    │   ├── connect_3.sh
    │   ├── connect_4.sh
    │   └── connect_5.sh
    └── vnclocs
        ├── connect_1.vncloc
        ├── connect_2.vncloc
        ├── connect_3.vncloc
        ├── connect_4.vncloc
        └── connect_5.vncloc

I previously wrote about JustTweak here. It's the feature flagging mechanism we've been using at Just Eat Takeaway.com to power the iOS consumer apps since 2017. It's proved to be very stable and powerful and it has evolved over time. Friends have heard me promoting it vehemently and some have integrated it with success and appreciation. I don't think I've promoted it in the community enough (it definitely deserved more) but marketing has never been my thing.

Anyway, JustTweak grew old and some changes were debatable and not to my taste. I have then decided to use the knowledge of years of working on the feature flagging matter to give this project a new life by rewriting it from scratch as a personal project.

And here it is: Toggles.

Think of JustTweak, but better. A lot better. Frankly, I couldn't have written it better. Here are the main highlights:

• brand new code, obsessively optimized and kept as short and simple as possible
• extreme performances
• fully tested
• fully documented
• performant UI debug view in SwiftUI
• standard providers provided
• demo app provided
• ability to listen for value changes (using Combine)
• simpler APIs
• ToggleGen CLI, to allow code generation
• ToggleCipher CLI, to allow encoding/decoding of secrets
• JustTweakMigrator CLI, to allow a smooth transition from JustTweak

Read all about it on the repo's README and on the DocC page.

It's on Swift Package Index too.

Toggles – Swift Package Index

Toggles by TogglesPlatform on the Swift Package Index – Toggles is an elegant and powerful solution to feature flagging for Apple platforms.

Learn more

There are plans (or at least the desire!) to write a backend with Andrea Scuderi. That'd be really nice!

Originally published on the Just Eat Takeaway Engineering Blog.

Overview

In this article, we’ll discuss the way our mobile teams have evolved the Continuous Integration (CI) stack over the recent years. We don’t have DevOps engineers in our team and, until recently, we had adopted a singular approach in which CI belongs to the whole team and everyone should be able to maintain it. This has proven to be difficult and extremely time-consuming.

The Just Eat side of our newly merged entity has a dedicated team providing continuous integration and deployment tools to their teams but they are heavily backend-centric and there has been little interest in implementing solutions tailored for mobile teams. As is often the case in tech companies, there is a missing link between mobile and DevOps teams.

The iOS team is the author and first consumer of the solution described but, as you can see, we have ported the same stack to Android as well. We will mainly focus on the iOS implementation in this article, with references to Android as appropriate.

2016–2020

Historically speaking, the iOS UK app was running on Bitrise because it was decided not to invest time in implementing a CI solution, while the Bristol team was using a Jenkins version installed by a different team. This required manual configuration with custom scripts and it had custom in-house hardware. These are two quite different approaches indeed and, at this stage, things were not great but somehow good enough. It’s fair to say we were still young on the DevOps front.

When we merged the teams, it was clear that we wanted to unify the CI solution and the obvious choice for a company of our size was to not use a third-party service, bringing us to invest more and more in Jenkins. Only one team member had good knowledge of Jenkins but the rest of the team showed little interest in learning how to configure and maintain it, causing the stack to eventually become a dumping ground of poorly configured jobs.

It was during this time that we introduced Fastlane (making the common tasks portable), migrated the UK app from Bitrise to Jenkins, started running the UI tests on Pull Requests, and other small yet sensible improvements.

2020–2021

Starting in mid-2020 the iOS team has significantly revamped its CI stack and given it new life. The main goals we wanted to achieve (and did by early 2021) were:

• Revisit the pipelines
• Clear Jenkins configuration and deployment strategy
• Make use of AWS Mac instances
• Reduce the pool size of our mac hardware
• Share our knowledge across teams better

Since the start of the pandemic, we have implemented the pipelines in code (bidding farewell to custom bash scripts), we moved to a monorepo which was a massive step ahead and began using SonarQube even more aggressively.

We added Slack reporting and PR Assigner, an internal tool implemented by Andrea Antonioni. We also automated the common release tasks such as cutting and completing a release and uploading the dSYMS to Firebase.

We surely invested a lot in optimizing various aspects such as running the UI tests in parallel, making use of shallow repo cloning, We also moved to not checking in the pods within the repo. This, eventually, allowed us to reduce the number of agents for easier infrastructure maintenance.

Automating the infrastructure deployment of Jenkins was a fundamental shift compared to the previous setup and we have introduced AWS Mac instances replacing part of the fleet of our in-house hardware.

CI system setup

Let’s take a look at our stack. Before we start, we’d like to thank Isham Araia for having provided a proof of concept for the configuration and deployment of Jenkins. He talked about it at https://ish-ar.io/jenkins-at-scale/ and it represented a fundamental starting point, saving us several days of researching.

Triggering flow

Starting from the left, we have our repositories (plural, as some shared dependencies don’t live in the monorepo). The repositories contain the pipelines in the form of Jenkinsfiles and they call into Fastlane lanes. Pretty much every action is a lane, from running the tests to archiving for the App Store to creating the release branches.

Changes are raised through pull requests that trigger Jenkins. There are other ways to trigger Jenkins: by user interaction (for things such as completing a release or archiving and uploading the app to App Store Connect) and cron triggers (for things such as building the nightly build, running the tests on the develop branch every 12 hours, or uploading the PACT contract to the broker).

Once Jenkins has received the information, it will then schedule the jobs to one of the agents in our pool, which is now made up of 5 agents, 2 in the cloud and 3 in-house mac pros.

Reporting flow

Now that we’ve talked about the first part of the flow, let’s talk about the flow of information coming back at us.

Every PR triggers PR Assigner, a tool that works out a list of reviewers to assign to pull requests and notifies engineers via dedicated Slack channels. The pipelines post on Slack, providing info about all the jobs that are being executed so we can read the history without having to log into Jenkins. We have in place the standard notification flow from Jenkins to GitHub to set the status checks and Jenkins also notifies SonarQube to verify that any change meets the quality standards (namely code coverage percentage and coding rules).

We also have a smart lambda named SonarQubeStatusProcessor that reports to GitHub, written by Alan Nichols. This is due to a current limitation of SonarQube, which only allows reporting the status of one SQ project to one GitHub repo. Since we have a monorepo structure we had to come up with this neat customization to report the SQ status for all the modules that have changed as part of the PR.

Configuration

Let’s see what the new interesting parts of Jenkins are. Other than Jenkins itself and several plugins, it’s important to point out JCasC and Job DSL.

JCasC stands for Jenkins Configuration as Code, and it allows you to configure Jenkins via a yaml file.

The point here is that nobody should ever touch the Jenkins settings directly from the configuration page, in the same way, one ideally shouldn’t apply configuration changes manually in any dashboard. The CasC file is where we define the Slack integration, the user roles, SSO configuration, the number of agents and so on.

We could also define the jobs in CasC but we go a step further than that.

We use the Job DSL plugin that allows you to configure the jobs in groovy and in much more detail. One job we configure in the CasC file though is the seed job. This is a simple freestyle job that will go pick the jobs defined with Job DSL and create them in Jenkins.

Deployment

Let’s now discuss how we can get a configured Jenkins instance on EC2. In other words, how do we deploy Jenkins?

We use a combination of tools that are bread and butter for DevOps people.

The commands on the left spawn a Docker container that calls into the tools on the right.

We start with Packer which allows us to create the AMI (Amazon Machine Image) together with Ansible, allowing us to configure an environment easily (much more easily than Chef or Puppet).

Running the create-image command the script will:

1. Create a temporary EC2 instance

2. Connect to the instance and execute an ansible playbook

Our playbook encompasses a number of steps, here’s a summary:

• install the Jenkins version for the given Linux distribution
• install Nginx
• copy the SSL cert over
• configure nginx w/ SSL termination and reverse proxy
• install the plugins for Jenkins

Once the playbook is executed, Packer will export an AMI in EC2 with all of this in it and destroy the instance that was used.

With the AMI ready, we can now proceed to deploy our Jenkins. For the actual deployment, we use Terraform which allows us to define our infrastructure in code.

The deploy command runs Terraform under the hood to set up the infrastructure, here’s a summary of the task:

• create an IAM Role + IAM Policy
• configure security groups
• create the VPC and subnet to use with a specific CIDER block and the subnet
• create any private key pair to connect over SSH
• deploy the instance using a static private IP (it has to be static otherwise the A record in Route53 would break)
• copy the JCasC configuration file over so that when Jenkins starts it picks that up to configure itself

The destroy command runs a “terraform destroy” and destroys everything that was created with the deploy command. Deploy and destroy balance each other out.

Now that we have Jenkins up and running, we need to give it some credentials so our pipelines are able to work properly. A neat way of doing this is by having the secrets (SSH keys, Firebase tokens, App Store Connect API Key and so forth) in AWS Secrets Manager which is based on KMS and use a Jenkins plugin to allow Jenkins to access them.

It’s important to note that developers don’t have to install Packer, Ansible, Terraform or even the AWS CLI locally because the commands run a Docker container that does the real work with all the tools installed. As a result, the only thing one should have installed is really Docker.

CI agents

Enough said about Jenkins, it’s time to talk about the agents.As you probably already know, in order to run tests, compile and archive iOS apps we need Xcode, which is only available on macOS, so Linux or Windows instances are not going to cut it.

We experimented with the recently introduced AWS Mac instances and they are great, ready out-of-the-box with minimal configuration on our end.

What we were hoping to get to with this recent work was the ability to leverage the Jenkins Cloud agents. That would have been awesome because it would have allowed us to:

• let Jenkins manage the agent instances
• scale the agent pool according to the load on CI

Sadly we couldn't go that far. Limitations are:

• the bootstrapping of a mac1.metal takes around 15 minutes
• reusing the dedicated host after having stopped an instance can take up to 3 hours — during that time we just pay for a host that is not usable

When you stop or terminate a Mac instance, Amazon EC2 performs a scrubbing workflow on the underlying Dedicated Host to erase the internal SSD, to clear the persistent NVRAM variables, and if needed, to update the bridgeOS software on the underlying Mac mini.

This ensures that Mac instances provide the same security and data privacy as other EC2 Nitro instances. It also enables you to run the latest macOS AMIs without manually updating the bridgeOS software. During the scrubbing workflow, the Dedicated Host temporarily enters the pending state. If the bridgeOS software does not need to be updated, the scrubbing workflow takes up to 50 minutes to complete. If the bridgeOS software needs to be updated, the scrubbing workflow can take up to 3 hours to complete.

Source: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-mac-instances.html

In other words: scaling mac instances is not an option and leaving the instances up 24/7 seems to be the easiest option. This is especially valid if your team is distributed and jobs could potentially run over the weekend as well, saving you the hassle of implementing downscaling ahead of the weekend.

There are some pricing and instance allocation considerations to make. Note that On-Demand Mac1 Dedicated Hosts have a minimum host allocation and billing duration of 24 hours.

“You can purchase Savings Plans to lower your spend on Dedicated Hosts. Savings Plans is a flexible pricing model that provides savings of up to 72% on your AWS compute usage. This pricing model offers lower prices on Amazon EC2 instances usage, regardless of instance family, size, OS, tenancy or AWS Region.”

Source: https://aws.amazon.com/ec2/dedicated-hosts/pricing/

The On-Demand rate is $1.207 per hour.

I’d like to stress that no CI solution comes for free. I’ve often heard developers indicating that Travis and similar products are cheaper. The truth is that the comparison is not even remotely reasonable: virtual boxes are incredibly slow compared to native Apple hardware and take ridiculous bootstrapping times. Even the smallest projects suffer terribly.

One might ask if it’s at least possible to use the same configuration process we used for the Jenkins instance (with Packer and Ansible) but sadly we hit additional limitations:

• Apple requires 2FA for downloading Xcode via xcode-version
• Apple requires 2FA for signing into Xcode

The above pretty much causes the configuration flow to fall apart making it impossible to configure an instance via Ansible.

Cloud agents for Android

It was a different story for Android, in which we could easily configure the agent instance with Ansible and therefore leverage the Cloud configuration to allow automatic agent provisioning.

This configuration is defined via CasC as everything else.

To better control EC2 usage and costs, a few settings come in handy:

• minimum number of instances (up at all times)
• minimum number of spare instances (created to accommodate future jobs)
• instance cap: the maximum number of instances that can be provisioned at the same time
• idle termination time: how long agents should be kept alive after they have completed the job

All of the above allow for proper scaling and a lot less maintenance compared to the iOS setup. A simple setup with 0 instances up at all times allows saving costs overnight and given that in our case the bootstrapping takes only 2 minutes, we can rely on the idle time setting.

Conclusions

Setting up an in-house CI is never a straightforward process and it requires several weeks of dedicated work.

After years of waiting, Apple has announced Xcode Cloud which we believe will drastically change the landscape of continuous integration on iOS. The solution will most likely cause havoc for companies such as Bitrise and CircleCI and it’s reasonable to assume the pricing will be competitive compared to AWS, maybe running on custom hardware that only Apple is able to produce.

A shift this big will take time to occur, so we foresee our solution to stay in use for quite some time.

We hope to have inspired you on how a possible setup for mobile teams could be and informed you on what are the pros & cons of using EC2 mac instances.

Originally published on the Just Eat Takeaway Engineering Blog.

We have presented our modular iOS architecture in a previous article and I gave a talk at Swift Heroes 2020 about it. In this article, we’ll analyse the challenges we faced to have the modular architecture integrated with our CI pipelines and the reasoning behind migrating to a monorepo.

The Problem

Having several modules in separate repositories brings forward 2 main problems:

1. Each module is versioned independently from the consuming app
2. Each change involves at least 2 pull requests: 1 for the module and 1 for the integration in the app

While the above was acceptable in a world where we had 2 different codebases, it soon became unnecessarily convoluted after we migrated to a new, global codebase. New module versions are implemented with the ultimate goal of being adopted by the only global codebase in use, making us realise we could simplify the change process.

The monorepo approach has been discussed at length by the community for a few years now. Many talking points have come out of these conversations, even leading to an interesting story as told by Uber. In short, it entails putting all the code owned by the team in a single repository, precisely solving the 2 problems stated above.

Monorepo structure

The main advantage of a monorepo is a streamlined PR process that doesn’t require us to raise multiple PRs, de facto reducing the number of pull requests to one.

It also simplifies the versioning, allowing module and app code (ultimately shipped together) to be aligned using the same versioning.

The first step towards a monorepo was to move the content of the repositories of the modules to the main app repo (we’ll call it “monorepo” from now on). Since we rely on CocoaPods, the modules would be consumed as development pods.

Here’s a brief summary of the steps used to migrate a module to the monorepo:

• Inform the relevant teams about the upcoming migration
• Make sure there are no open PRs in the module repo
• Make the repository read-only and archive it
• Copy the module to the Modules folder of the monorepo (it’s possible to merge 2 repositories to keep the history but we felt we wanted to keep the process simple, the old history is still available in the old repo anyway)
• Delete the module .git folder (or it would cause a git submodule)
• Remove Gemfile and Gemfile.lock fastlane folder, .gitignore file, sonar-project.properties, .swiftlint.yml so to use those in the monorepo
• Update the monorepo’s CODEOWNERS file with the module codeowners
• Remove the .github folder
• Modify the app Podfile to point to the module as a dev pod and install it
• Make sure all the modules’ demo apps in the monorepo refer to the new module as a dev pod (if they depend on it at all). The same applies to the module under migration.
• Delete the CI jobs related to the module
• Leave the podspecs in the private Specs repo (might be needed to build old versions of the app)

The above assumes that CI is configured in a way that preserves the same integration steps upon a module change. We’ll discuss them later in this article.

Not all the modules could be migrated to the monorepo, due to the fact the second-level dependencies need to live in separate repositories in order to be referenced in the podspec of a development pod. If not done correctly, CocoaPods will not be able to install them. We considered moving these dependencies to the monorepo whilst maintaining separate versioning, however, the main problem with this approach is that the version tags might conflict with the ones of the app. Even though CocoaPods supports tags that don’t respect semantic versioning (for example prepending the tag with the name of the module), violating it just didn’t feel right.

EDIT: we’ve learned that it’s possible to move such dependencies to the monorepo. This is done not by defining :path=> in the podspecs but instead by doing so in the Podfile of the main app, which is all Cocoapods needs to work out the location of the dependency on disk.

Swift Package Manager considerations

We investigated the possibility of migrating from CocoaPods to Apple’s Swift Package Manager. Unfortunately, when it comes to handling the equivalent of development pods, Swift Package Manager really falls down for us. It turns out that Swift Package Manager only supports one package per repo, which is frustrating because the process of working with editable packages is surprisingly powerful and transparent.

Version pinning rules

While development pods don’t need to be versioned, other modules still need to. This is either because of their open-source nature or because they are second-level dependencies (referenced in other modules’ podspecs).

Here’s a revised overview of the current modular architecture in 2021.

We categorised our pods to better clarify what rules should apply when it comes to version pinning both in the Podfiles and in the podspecs.

Open-Source pods

Our open-source repositories on github.com/justeat are only used by the app.

• Examples: JustTweak, AutomationTools, Shock
• Pinning in other modules’ podspec: NOT APPLICABLE open-source pods don’t appear in any podspec, those that do are called ‘open-source shared’
• Pinning in other modules’ Podfile (demo apps): PIN (e.g. AutomationTools in Orders demo app’s Podfile)
• Pinning in main app’s Podfile: PIN (e.g. AutomationTools)

Open-Source shared pods

The Just Eat pods we put open-source on github.com/justeat and are used by modules and apps.

• Examples: JustTrack, JustLog, ScrollingStackViewController, ErrorUtilities
• Pinning in other modules’ podspec: PIN w/ optimistic operator (e.g. JustTrack in Orders)
• Pinning in other modules’ Podfile (demo apps): PIN (e.g. JustTrack in Orders demo app’s Podfile)
• Pinning in main app’s Podfile: DON’T LIST latest compatible version is picked by CocoaPods (e.g. JustTrack). LIST & PIN if the pod is explicitly used in the app too, so we don’t magically inherit it.

Internal Domain pods

Domain modules (yellow).

• Examples: Orders, SERP, etc.
• Pinning in other modules’ podspec: NOT APPLICABLE domain pods don’t appear in other pods’ podspecs (domain modules don’t depend on other domain modules)
• Pinning in other modules’ Podfile (demo apps): PIN only if the pod is used in the app code, rarely the case (e.g. Account in Orders demo app’s Podfile)
• Pinning in main app’s Podfile: PIN (e.g. Orders)

Internal Core pods

Core modules (blue) minus those open-source.

• Examples: APIClient, AssetProvider
• Pinning in other modules’ podspec: NOT APPLICABLE core pods don’t appear in other pods’ podspecs (core modules are only used in the app(s))
• Pinning in other modules’ Podfile (demo apps): PIN only if pod is used in the app code (e.g. APIClient in Orders demo app’s Podfile)
• Pinning in main app’s Podfile: PIN (e.g. NavigationEngine)

Internal shared pods

Shared modules (green) minus those open-source.

• Examples: JustUI, JustAnalytics
• Pinning in other modules’ podspec: DON’T PIN (e.g. JustUI in Orders podspec)
• Pinning in other modules’ Podfile (demo apps): PIN (e.g. JustUI in Orders demo app’s Podfile)
• Pinning in main app’s Podfile: PIN (e.g. JustUI)

External shared pods

Any non-Just Eat pod used by any internal or open-source pod.

• Examples: Usabilla, SDWebImage
• Pinning in other modules’ podspec: PIN (e.g. Usabilla in Orders)
• Pinning in other modules’ Podfile (demo apps): DON’T LIST because the version is forced by the podspec. LIST & PIN if the pod is explicitly used in the app too, so we don’t magically inherit it. Pinning is irrelevant but good practice.
• Pinning in main app’s Podfile: DON’T LIST because the version is forced by the podspec(s). LIST & PIN if the pod is explicitly used in the app too, so we don’t magically inherit it. Pinning is irrelevant but good practice.

External pods

Any non-Just Eat pod used by the app only.

• Examples: Instabug, GoogleAnalytics
• Pinning in other modules’ podspec: NOT APPLICABLE external pods don’t appear in any podspec, those that do are called ‘external shared’
• Pinning in other modules’ Podfile (demo apps): PIN only if the pod is used in the app code, rarely the case (e.g. Promis)
• Pinning in main app’s Podfile: PIN (e.g. Adjust)

Pinning is a good solution because it guarantees that we always build the same software regardless of new released versions of dependencies. It’s also true that pinning every dependency all the time makes the dependency graph hard to keep updated. This is why we decided to allow some flexibility in some cases.

Following is some more reasoning.

Open-source

For “open-source shared” pods, we are optimistic enough (pun intended) to tolerate the usage of the optimistic operator ~> in podspecs of other pods (i.e Orders using JustTrack) so that when a new patch version is released, the consuming pod gets it for free upon running pod update.

We have control over our code and, by respecting semantic versioning, we guarantee the consuming pod to always build. In case of new minor or major versions, we would have to update the podspecs of the consuming pods, which is appropriate.

Also, we do need to list any “open-source shared” pod in the main app’s Podfile only if directly used by the app code.

External

We don’t have control over the “external” and “external shared” pods, therefore we always pin the version in the appropriate place. New patch versions might not respect semantic versioning for real and we don’t want to pull in new code unintentionally. As a rule of thumb, we prefer injecting external pods instead of creating a dependency in the podspec.

Internal

Internal shared pods could change frequently (not as much as domain modules). For this reason, we’ve decided to relax a constraint we had and not to pin the version in the podspec. This might cause the consuming pod to break when a new version of an “internal shared” pod is released and we run pod update. This is a compromise we can tolerate. The alternative would be to pin the version causing too much work to update the podspec of the domain modules.

Continuous Integration changes

With modules in separate repositories, the CI was quite simply replicating the same steps for each module:

• install pods
• run unit tests
• run UI tests
• generated code coverage
• submit code coverage to SonarQube

Moving the modules to the monorepo meant creating smart CI pipelines that would run the same steps upon modules’ changes.

If a pull request is to change only app code, there is no need to run any step for the modules, just the usual steps for the app:

If instead, a pull request applies changes to one or more modules, we want the pipeline to first run the steps for the modules, and then the steps for the app:

Even if there are no changes in the app code, module changes could likely impact the app behaviour, so it’s important to always run the app tests.

We have achieved the above setup through constructing our Jenkins pipelines dynamically. The solution should scale when new modules are added to the monorepo and for this reason, it’s important that all modules:

• respect the same project setup (generated by CocoaPods w/ the pod lib create command)
• use the same naming conventions for the test schemes (UnitTests/ContractTests/UITests)
• make use of Apple Test Plans
• are in the same location ( ./Modules/ folder).

Following is an excerpt of the code that constructs the modules’ stages from the Jenkinsfile used for pull request jobs.

scripts = load "./Jenkins/scripts/scripts.groovy"

def modifiedModules = scripts.modifiedModulesFromReferenceBranch(env.CHANGE_TARGET)

def modulesThatNeedUpdating = scripts.modulesThatNeedUpdating(env.CHANGE_TARGET)

def modulesToRun = (modulesThatNeedUpdating + modifiedModules).unique()

sh "echo \"List of modules modified on this branch: ${modifiedModules}\""

sh "echo \"List of modules that need updating: ${modulesThatNeedUpdating}\""

sh "echo \"Pipeline will run the following modules: ${modulesToRun}\""

for (int i = 0; i < modulesToRun.size(); ++i) {
    def moduleName = modulesToRun[i]
    stage('Run pod install') {
        sh "bundle exec fastlane pod_install module:${moduleName}"
    }

    def schemes = scripts.testSchemesForModule(moduleName)
    schemes.each { scheme ->
        switch (scheme) {
            case "UnitTests":
                stage("${moduleName} Unit Tests") {
                    sh "bundle exec fastlane module_unittests \
                    module_name:${moduleName} \
                    device:'${env.IPHONE_DEVICE}'"
                }
                stage("Generate ${moduleName} code coverage") {
                    sh "bundle exec fastlane generate_sonarqube_coverage_xml"
                }
                stage("Submit ${moduleName} code coverage to SonarQube") {
                    sh "bundle exec fastlane sonar_scanner_pull_request \
                    component_type:'module' \
                    source_branch:${env.BRANCH_NAME} \
                    target_branch:${env.CHANGE_TARGET} \
                    pull_id:${env.CHANGE_ID} \
                    project_key:'ios-${moduleName}' \
                    project_name:'iOS ${moduleName}' \
                    sources_path:'./Modules/${moduleName}/${moduleName}'"
                }
                break;
            case "ContractTests":
                stage('Install pact mock service') {
                    sh "bundle exec fastlane install_pact_mock_service"
                }
                stage("${moduleName} Contract Tests") {
                    sh "bundle exec fastlane module_contracttests \
                    module_name:${moduleName} \
                    device:'${env.IPHONE_DEVICE}'"
                }
                break;
            case "UITests":
                stage("${moduleName} UI Tests") {
                    sh "bundle exec fastlane module_uitests \
                    module_name:${moduleName} \
                    number_of_simulators:${env.NUMBER_OF_SIMULATORS} \
                    device:'${env.IPHONE_DEVICE}'"
                }
                break;
            default: break;
        }
    }
}

and here are the helper functions to make it all work:

def modifiedModulesFromReferenceBranch(String referenceBranch) {
    def script = "git diff --name-only remotes/origin/${referenceBranch}"
    def filesChanged = sh script: script, returnStdout: true
    Set modulesChanged = []
    filesChanged.tokenize("\n").each {
        def components = it.split('/')
        if (components.size() > 1 && components[0] == 'Modules') { 
            def module = components[1]
            modulesChanged.add(module)
        }
    }
    return modulesChanged
}

def modulesThatNeedUpdating(String referenceBranch) {
    def modifiedModules = modifiedModulesFromReferenceBranch(referenceBranch)
    def allModules = allMonorepoModules()
    def modulesThatNeedUpdating = []
    for (module in allModules) {
        def podfileLockPath = "Modules/${module}/Example/Podfile.lock"
        def dependencies = podfileDependencies(podfileLockPath)
        def dependenciesIntersection = dependencies.intersect(modifiedModules) as TreeSet
        Boolean moduleNeedsUpdating = (dependenciesIntersection.size() > 0)
        if (moduleNeedsUpdating == true && modifiedModules.contains(module) == false) {
            modulesThatNeedUpdating.add(module)
        }
    }
    return modulesThatNeedUpdating
}

def podfileDependencies(String podfileLockPath) {
    def dependencies = []
    def fileContent = readFile(file: podfileLockPath)
    fileContent.tokenize("\n").each { line ->
        def lineComponents = line.split('\\(')
        if (lineComponents.length > 1) {
            def dependencyLineSubComponents = lineComponents[0].split('-')
            if (dependencyLineSubComponents.length > 1) {
                def moduleName = dependencyLineSubComponents[1].trim()
                dependencies.add(moduleName)
            }
        }
    }
    return dependencies
}

def allMonorepoModules() {
    def modulesList = sh script: "ls Modules", returnStdout: true
    return modulesList.tokenize("\n").collect { it.trim() }
}

def testSchemesForModule(String moduleName) {
    def script = "xcodebuild -project ./Modules/${moduleName}/Example/${moduleName}.xcodeproj -list"
    def projectEntitites = sh script: script, returnStdout: true
    def schemesPart = projectEntitites.split('Schemes:')[1]
    def schemesPartLines = schemesPart.split(/\n/)
    def trimmedLined = schemesPartLines.collect { it.trim() }
    def filteredLines = trimmedLined.findAll { !it.allWhitespace }
    def allowedSchemes = ['UnitTests', 'ContractTests', 'UITests']
    def testSchemes = filteredLines.findAll { allowedSchemes.contains(it) }
    return testSchemes
}

You might have noticed the modulesThatNeedUpdating method in the code above. Each module comes with a demo app using the dependencies listed in its Podfile and it’s possible that other monorepo modules are listed there as development pods. This not only means that we have to run the steps for the main app, but also the steps for every module consuming modules that show changes.

For example, the Orders demo app uses APIClient, meaning that pull requests with changes in APIClient will generate pipelines including the Orders steps.

Pipeline parallelization

Something we initially thought was sensible to consider is the parallelisation of the pipelines across different nodes. We use parallelisation for the release pipelines and learned that, while it seems to be a fundamental requirement at first, it soon became apparent not to be so desirable nor truly fundamental for the pull requests pipeline.

We’ll discuss our CI setup in a separate article, but suffice to say that we have aggressively optimized it and managed to reduce the agent pool from 10 to 5, still maintaining a good level of service.

Parallelisation sensibly complicates the Jenkinsfiles and their maintainability, spreads the cost of checking out the repository across nodes and makes the logs harder to read. The main benefit would come from running the app UI tests on different nodes. In the WWDC session 413, Apple recommends generating the .xctestrun file using the build-for-testing option in xcodebuild and distribute it across the other nodes. Since our app is quite large, such file is also large and transferring it has its costs, both in time and bandwidth usage.

All things considered, we decided to keep the majority of our pipelines serial.

EDIT: In 2022 we have parallelised our PR pipeline in 4 branches:

• Validation steps (linting, Fastlane lanes tests, etc.)
• App unit tests
• App UI tests (short enough that there's no need to share .xctestrun across nodes)
• Modified modules unit tests
• Modified modules UI tests

Conclusions

We have used the setup described in this article since mid-2020 and we are very satisfied with it. We discussed the pipeline used for the pull requests which is the most relevant one when it comes to embracing a monorepo structure. We have a few more pipelines for various use cases, such as verifying changes in release branches, keeping the code coverage metrics up-to-date with jobs running of triggers, archiving the app for internal usage and for App Store.

We hope to have given you some useful insights on how to structure a monorepo and its CI pipelines, especially if you have a structure similar to ours.

I wrote the first version of iHarmony in 2008. It was the very first iOS app I gave birth to, combining my passion for music and programming. I remember buying an iPhone and my first Mac with the precise purpose of jumping on the apps train at a time when it wasn't clear if the apps were there to stay or were just a temporary hype. But I did it, dropped my beloved Ubuntu to join a whole new galaxy. iHarmony was also one of the first 2000 apps on the App Store.

Up until the recent rewrite, iHarmony was powered by a manually crafted database containing scales, chords, and harmonization I inputted.

What-a-shame!

I guess it made sense, I wanted to learn iOS and not to focus on implementing some core logic independent from the platform. Clearly a much better and less error-prone way to go would be to implement an algorithm to generate all the entries based on some DSL/spec. It took me almost 12 years to decide to tackle the problem and I've recently realized that writing the algorithm I wanted was harder than I thought. Also thought was a good idea give SwiftUI a try since the UI of iHarmony is extremely simple but... nope.

Since someone on the Internet expressed interest 😉, I wrote this article to explain how I solved the problem of modeling music theory concepts in a way that allows the generation of any sort of scales, chords, and harmonization. I only show the code needed to get a grasp of the overall structure.

I know there are other solutions ready to be used on GitHub but, while I don't particularly like any of them, the point of rewriting iHarmony from scratch was to challenge myself, not to reuse code someone else wrote. Surprisingly to me, getting to the solution described here took me 3 rewrites and 2 weeks.

Solution

The first fundamental building blocks to model are surely the musical notes, which are made up of a natural note and an accidental.

enum NaturalNote: String {
    case C, D, E, F, G, A, B
}

enum Accidental: String {
    case flatFlatFlat = "bbb"
    case flatFlat = "bb"
    case flat = "b"
    case natural = ""
    case sharp = "#"
    case sharpSharp = "##"
    case sharpSharpSharp = "###"
    
    func applyAccidental(_ accidental: Accidental) throws -> Accidental {...}
}

struct Note: Hashable, Equatable {
    
    let naturalNote: NaturalNote
    let accidental: Accidental
    
    ...
    
    static let Dff = Note(naturalNote: .D, accidental: .flatFlat)
    static let Df = Note(naturalNote: .D, accidental: .flat)
    static let D = Note(naturalNote: .D, accidental: .natural)
    static let Ds = Note(naturalNote: .D, accidental: .sharp)
    static let Dss = Note(naturalNote: .D, accidental: .sharpSharp)
    
    ...
    
    func noteByApplyingAccidental(_ accidental: Accidental) throws -> Note {...}
}

Combinations of notes make up scales and chords and they are... many. What's fixed instead in music theory, and therefore can be hard-coded, are the keys (both major and minor) such as:

• C major: C, D, E, F, G, A, B
• A minor: A, B, C, D, E, F, G
• D major: D, E, F#, G, A, B, C#

We'll get back to the keys later, but we can surely implement the note sequence for each musical key.

typealias NoteSequence = [Note]

extension NoteSequence {
    static let C = [Note.C, Note.D, Note.E, Note.F, Note.G, Note.A, Note.B]
    static let A_min = [Note.A, Note.B, Note.C, Note.D, Note.E, Note.F, Note.G]
    
    static let G = [Note.G, Note.A, Note.B, Note.C, Note.D, Note.E, Note.Fs]
    static let E_min = [Note.E, Note.Fs, Note.G, Note.A, Note.B, Note.C, Note.D]
    
    ...
}

Next stop: intervals. They are a bit more interesting as not every degree has the same types. Let's split into 2 sets:

1. 2nd, 3rd, 6th and 7th degrees can be minor, major, diminished and augmented
2. 1st (and 8th), 4th and 5th degrees can be perfect, diminished and augmented.

We need to use different kinds of "diminished" and "augmented" for the 2 sets as later on we'll have to calculate the accidentals needed to turn an interval into another.

Some examples:

• to get from 2nd augmented to 2nd diminished, we need a triple flat accidental (e.g. in C major scale, from D♯ to D♭♭ there are 3 semitones)
• to get from 5th augmented to 5th diminished, we need a double flat accidental (e.g. in C major scale, from G♯ to G♭there are 2 semitones)

We proceed to hard-code the allowed intervals in music, leaving out the invalid ones (e.g. Interval(degree: ._2, type: .augmented))

enum Degree: Int, CaseIterable {
    case _1, _2, _3, _4, _5, _6, _7, _8
}

enum IntervalType: Int, RawRepresentable {
    case perfect
    case minor
    case major
    case diminished
    case augmented
    case minorMajorDiminished
    case minorMajorAugmented
}

struct Interval: Hashable, Equatable {
    let degree: Degree
    let type: IntervalType
    
    static let _1dim = Interval(degree: ._1, type: .diminished)
    static let _1    = Interval(degree: ._1, type: .perfect)
    static let _1aug = Interval(degree: ._1, type: .augmented)
    
    static let _2dim = Interval(degree: ._2, type: .minorMajorDiminished)
    static let _2min = Interval(degree: ._2, type: .minor)
    static let _2maj = Interval(degree: ._2, type: .major)
    static let _2aug = Interval(degree: ._2, type: .minorMajorAugmented)
    
    ...
    
    static let _4dim = Interval(degree: ._4, type: .diminished)
    static let _4    = Interval(degree: ._4, type: .perfect)
    static let _4aug = Interval(degree: ._4, type: .augmented)
    
    ...
    
    static let _7dim = Interval(degree: ._7, type: .minorMajorDiminished)
    static let _7min = Interval(degree: ._7, type: .minor)
    static let _7maj = Interval(degree: ._7, type: .major)
    static let _7aug = Interval(degree: ._7, type: .minorMajorAugmented)
}

Now it's time to model the keys (we touched on them above already). What's important is to define the intervals for all of them (major and minor ones).

enum Key {
    // natural
    case C, A_min
    
    // sharp
    case G, E_min
    case D, B_min
    case A, Fs_min
    case E, Cs_min
    case B, Gs_min
    case Fs, Ds_min
    case Cs, As_min
    
    // flat
    case F, D_min
    case Bf, G_min
    case Ef, C_min
    case Af, F_min
    case Df, Bf_min
    case Gf, Ef_min
    case Cf, Af_min
    
    ...
    
    enum KeyType {
        case naturalMajor
        case naturalMinor
        case flatMajor
        case flatMinor
        case sharpMajor
        case sharpMinor
    }
    
    var type: KeyType {
        switch self {
        case .C: return .naturalMajor
        case .A_min: return .naturalMinor
        case .G, .D, .A, .E, .B, .Fs, .Cs: return .sharpMajor
        case .E_min, .B_min, .Fs_min, .Cs_min, .Gs_min, .Ds_min, .As_min: return .sharpMinor
        case .F, .Bf, .Ef, .Af, .Df, .Gf, .Cf: return .flatMajor
        case .D_min, .G_min, .C_min, .F_min, .Bf_min, .Ef_min, .Af_min: return .flatMinor
        }
    }
    
    var intervals: [Interval] {
        switch type {
        case .naturalMajor, .flatMajor, .sharpMajor:
            return [
                ._1, ._2maj, ._3maj, ._4, ._5, ._6maj, ._7maj
            ]
        case .naturalMinor, .flatMinor, .sharpMinor:
            return [
                ._1, ._2maj, ._3min, ._4, ._5, ._6min, ._7min
            ]
        }
    }
    
    var notes: NoteSequence {
        switch self {
        case .C: return .C
        case .A_min: return .A_min
    	...
    }
}

At this point we have all the fundamental building blocks and we can proceed with the implementation of the algorithm.

The idea is to have a function that given

• a key
• a root interval
• a list of intervals

it works out the list of notes. In terms of inputs, it seems the above is all we need to correctly work out scales, chords, and - by extension - also harmonizations. Mind that the root interval doesn't have to be part of the list of intervals, that is simply the interval to start from based on the given key.

Giving a note as a starting point is not good enough since some scales simply don't exist for some notes (e.g. G♯ major scale doesn't exist in the major key, and G♭minor scale doesn't exist in any minor key).

Before progressing to the implementation, please consider the following unit tests that should make sense to you:

func test_noteSequence_C_1() {
    let key: Key = .C
    let noteSequence = try! engine.noteSequence(customKey: key.associatedCustomKey,
                                                intervals: [._1, ._2maj, ._3maj, ._4, ._5, ._6maj, ._7maj])
    let expectedValue: NoteSequence = [.C, .D, .E, .F, .G, .A, .B]
    XCTAssertEqual(noteSequence, expectedValue)
}
    
func test_noteSequence_withRoot_C_3maj_majorScaleIntervals() {
    let key = Key.C
    let noteSequence = try! engine.noteSequence(customKey: key.associatedCustomKey,
                                                rootInterval: ._3maj,
                                                intervals: [._1, ._2maj, ._3maj, ._4, ._5, ._6maj, ._7maj])
    let expectedValue: NoteSequence = [.E, .Fs, .Gs, .A, .B, .Cs, .Ds]
    XCTAssertEqual(noteSequence, expectedValue)
}
    
func test_noteSequence_withRoot_Gsmin_3maj_alteredScaleIntervals() {
    let key = Key.Gs_min
    let noteSequence = try! engine.noteSequence(customKey: key.associatedCustomKey,
                                                rootInterval: ._3maj,
                                                intervals: [._1aug, ._2maj, ._3dim, ._4dim, ._5aug, ._6dim, ._7dim])
    let expectedValue: NoteSequence = [.Bs, .Cs, .Df, .Ef, .Fss, .Gf, .Af]
    XCTAssertEqual(noteSequence, expectedValue)
}

and here is the implementation. Let's consider a simple case, so it's easier to follow:

• key = C major
• root interval = 3maj
• interval = major scale interval (1, 2maj, 3min, 4, 5, 6maj, 7min)

if you music theory allowed you to understand the above unit tests, you would expect the output to be: E, F♯, G, A, B, C♯, D (which is a Dorian scale).

Steps:

1. we start by shifting the notes of the C key to position the 3rd degree (based on the 3maj) as the first element of the array, getting the note sequence E, F, G, A, B, C, D;
2. here's the first interesting bit: we then get the list of intervals by calculating the number of semitones from the root to any other note in the sequence and working out the corresponding Interval:
   1_perfect, 2_minor, 3_minor, 4_perfect, 5_perfect, 6_minor, 7_minor;
3. we now have all we need to create a CustomKey which is pretty much a Key (with notes and intervals) but instead of being an enum with pre-defined values, is a struct;
4. here's the second tricky part: return the notes by mapping the input intervals. Applying to each note in the custom key the accidental needed to match the desired interval. In our case, the only 2 intervals to 'adjust' are the 2nd and the 6th intervals, both minor in the custom key but major in the list of intervals. So we have to apply a sharp accidental to 'correct' them.

👀 I've used force unwraps in these examples for simplicity, the code might already look complex by itself.

class CoreEngine {

    func noteSequence(customKey: CustomKey,
                      rootInterval: Interval = ._1,
                      intervals: [Interval]) throws -> NoteSequence {
        // 1.
        let noteSequence = customKey.shiftedNotes(by: rootInterval.degree)
        let firstNoteInShiftedSequence = noteSequence.first!
        
        // 2.
        let adjustedIntervals = try noteSequence.enumerated().map {
            try interval(from: firstNoteInShiftedSequence,
                         to: $1,
                         targetDegree: Degree(rawValue: $0)!)
        }
        
        // 3.
        let customKey = CustomKey(notes: noteSequence,
                                  intervals: adjustedIntervals)
        
        // 4.
        return try intervals.map {
            let referenceInterval = customKey.firstIntervalWithDegree($0.degree)!
            let note = customKey.notes[$0.degree.rawValue]
            let accidental = try referenceInterval.type.accidental(to: $0.type)
            return try note.noteByApplyingAccidental(accidental)
        }
    }
}

It's worth showing the implementation of the methods used above:

private func numberOfSemitones(from sourceNote: Note,
                               to targetNote: Note) -> Int {
    let notesGroupedBySameTone: [[Note]] = [
        [.C, .Bs, .Dff],
        [.Cs, .Df, .Bss],
        [.D, .Eff, .Css],
        [.Ds, .Ef, .Fff],
        [.E, .Dss, .Ff],
        [.F, .Es, .Gff],
        [.Fs, .Ess, .Gf],
        [.G, .Fss, .Aff],
        [.Gs, .Af],
        [.A, .Gss, .Bff],
        [.As, .Bf, .Cff],
        [.B, .Cf, .Ass]
    ]
        
    let startIndex = notesGroupedBySameTone.firstIndex { $0.contains(sourceNote)}!
    let endIndex = notesGroupedBySameTone.firstIndex { $0.contains(targetNote)}!
        
    return endIndex >= startIndex ? endIndex - startIndex : (notesGroupedBySameTone.count - startIndex) + endIndex
}
    
private func interval(from sourceNote: Note,
                      to targetNote: Note,
                      targetDegree: Degree) throws -> Interval {
    let semitones = numberOfSemitones(from: sourceNote, to: targetNote)
        
    let targetType: IntervalType = try {
        switch targetDegree {
        case ._1, ._8:
            return .perfect
        ...
        case ._4:
            switch semitones {
            case 4:
                return .diminished
            case 5:
                return .perfect
            case 6:
                return .augmented
            default:
                throw CustomError.invalidConfiguration
        ...
        case ._7:
            switch semitones {
            case 9:
                return .minorMajorDiminished
            case 10:
                return .minor
            case 11:
                return .major
            case 0:
                return .minorMajorAugmented
            default:
                throw CustomError.invalidConfiguration
            }
        }
    }()
    return Interval(degree: targetDegree, type: targetType)
}

the Note's noteByApplyingAccidental method:

func noteByApplyingAccidental(_ accidental: Accidental) throws -> Note {
    let newAccidental = try self.accidental.apply(accidental)
    return Note(naturalNote: naturalNote, accidental: newAccidental)
}

and the Accidental's apply method:

func apply(_ accidental: Accidental) throws -> Accidental {
    switch (self, accidental) {
    ...
    case (.flat, .flatFlatFlat):
        throw CustomError.invalidApplicationOfAccidental
    case (.flat, .flatFlat):
        return .flatFlatFlat
    case (.flat, .flat):
        return .flatFlat
    case (.flat, .natural):
        return .flat
    case (.flat, .sharp):
        return .natural
    case (.flat, .sharpSharp):
        return .sharp
    case (.flat, .sharpSharpSharp):
        return .sharpSharp
            
    case (.natural, .flatFlatFlat):
        return .flatFlatFlat
    case (.natural, .flatFlat):
        return .flatFlat
    case (.natural, .flat):
        return .flat
    case (.natural, .natural):
        return .natural
    case (.natural, .sharp):
        return .sharp
    case (.natural, .sharpSharp):
        return .sharpSharp
    case (.natural, .sharpSharpSharp):
        return .sharpSharpSharp   
    ...
}

With the above engine ready (and 💯﹪ unit tested!), we can now proceed to use it to work out what we ultimately need (scales, chords, and harmonizations).

extension CoreEngine {
    func scale(note: Note, scaleIdentifier: Identifier) throws -> NoteSequence {...}
    func chord(note: Note, chordIdentifier: Identifier) throws -> NoteSequence {...}
    func harmonization(key: Key, harmonizationIdentifier: Identifier) throws -> NoteSequence {...}
    func chordSignatures(note: Note, scaleHarmonizationIdentifier: Identifier) throws -> [ChordSignature] {...}
    func harmonizations(note: Note, scaleHarmonizationIdentifier: Identifier) throws -> [NoteSequence] {...}
}

Conclusions

There's more to it but with this post I only wanted to outline the overall idea.

The default database is available on GitHub at albertodebortoli/iHarmonyDB. The format used is JSON and the community can now easily suggest additions.

Here is how the definition of a scale looks:

"scale_dorian": {
    "group": "group_scales_majorModes",
    "isMode": true,
    "degreeRelativeToMain": 2,
    "inclination": "minor",
    "intervals": [
        "1",
        "2maj",
        "3min",
        "4",
        "5",
        "6maj",
        "7min"
    ]
}

and a chord:

"chord_diminished": {
    "group": "group_chords_diminished",
    "abbreviation": "dim",
    "intervals": [
        "1",
        "3min",
        "5dim"
    ]
}

and a harmonization:

"scaleHarmonization_harmonicMajorScale4Tones": {
    "group": "group_harmonization_harmonic_major",
    "inclination": "major",
    "harmonizations": [
        "harmonization_1_major7plus",
        "harmonization_2maj_minor7dim5",
        "harmonization_3maj_minor7",
        "harmonization_4_minor7plus",
        "harmonization_5_major7",
        "harmonization_6min_major7plus5sharp",
        "harmonization_7maj_diminished7"
    ]
}

Have to say, I'm pretty satisfied with how extensible this turned out to be.

Thanks for reading 🎶

In this article, I try to summarize the bare minimum one needs to know to add internationalization support to an iOS app.

Localizations, locales, timezones, date and currency formatting... it's shocking how easy is to forget how they work and how to use them correctly.

After years more than 10 years into iOS development, I decided to write down a few notes on the matter, with the hope that they will come handy again in the future, hopefully not only to me.

TL;DR

From Apple docs:

Date: a specific point in time, independent of any calendar or time zone;

TimeZone: information about standard time conventions associated with a specific geopolitical region;

Locale: information about linguistic, cultural, and technological conventions for use in formatting data for presentation.

Rule of thumb:

• All DateFormatters should use the locale and the timezone of the device;
• All NumberFormatter, in particular those with numberStyle set to .currency (for the sake of this article) should use a specific locale so that prices are not shown in the wrong currency.

General notes on formatters

Let's start by stating the obvious.

Since iOS 10, Foundation (finally) provides ISO8601DateFormatter, which, alongside with DateFormatter and NumberFormatter, inherits from Formatter.

In an app that only consumes data from an API, the main purpose of ISO8601DateFormatter is to convert strings to dates (String -> Date) more than the inverse. DateFormatter is then used to format dates (Date -> String) to ultimately show the values in the UI. NumberFormatter instead, converts numbers (prices in the vast majority of the cases) to strings (NSNumber/Decimal -> String).

Formatting dates 🕗 🕝 🕟

It seems the following 4 are amongst the most common ISO 8601 formats, including the optional UTC offset.

• A: 2019-10-02T16:53:42
• B: 2019-10-02T16:53:42Z
• C: 2019-10-02T16:53:42-02:00
• D: 2019-10-02T16:53:42.974Z

In this article I'll stick to these formats.

The 'Z' at the end of an ISO8601 date indicates that it is in UTC, not a local time zone.

Locales

Converting strings to dates (String -> Date) is done using ISO8601DateFormatter objects set up with various formatOptions.

Once we have a Date object, we can deal with the formatting for the presentation. Here, the locale is important and things can get a bit tricky. Locales have nothing to do with timezones, locales are for applying a format using a language/region.

Locale identifiers are in the form of <language_identifier>_<region_identifier> (e.g. en_GB).

We should use the user's locale when formatting dates (Date -> String). Consider a British user moving to Italy, the apps should keep showing a UI localized in English, and the same applies to the dates that should be formatted using the en_GB locale. Using the it_IT locale would show "2 ott 2019, 17:53" instead of the correct "2 Oct 2019 at 17:53".

Locale.current, shows the locale set (overridden) in the iOS simulator and setting the language and regions in the scheme's options comes handy for debugging.

Some might think that it's acceptable to use Locale.preferredLanguages.first and create a Locale from it with let preferredLanguageLocale = Locale(identifier: Locale.preferredLanguages.first!) and set it on the formatters. I think that doing so is not great since we would display dates using the Italian format but we won't necessarily be using the Italian language for the other UI elements as the app might not have the IT localization, causing an inconsistent experience. In short: don't use preferredLanguages, best to use Locale.current.

Apple strongly suggests using en_US_POSIX pretty much everywhere (1, 2). From Apple docs:

[...] if you're working with fixed-format dates, you should first set the locale of the date formatter to something appropriate for your fixed format. In most cases the best locale to choose is "en_US_POSIX", a locale that's specifically designed to yield US English results regardless of both user and system preferences. "en_US_POSIX" is also invariant in time (if the US, at some point in the future, changes the way it formats dates, "en_US" will change to reflect the new behaviour, but "en_US_POSIX" will not), and between machines ("en_US_POSIX" works the same on iOS as it does on OS X, and as it it does on other platforms).

Once you've set "en_US_POSIX" as the locale of the date formatter, you can then set the date format string and the date formatter will behave consistently for all users.

I couldn't find a really valid reason for doing so and quite frankly using the device locale seems more appropriate for converting dates to strings.

Here is the string representation for the same date using different locales:

• en_US_POSIX: May 2, 2019 at 3:53 PM
• en_GB: 2 May 2019 at 15:53
• it_IT: 2 mag 2019, 15:53

The above should be enough to show that en_US_POSIX is not what we want to use in this case, but it has more to do with maintaining a standard for communication across machines. From this article:

"[...] Unless you specifically need month and/or weekday names to appear in the user's language, you should always use the special locale of en_US_POSIX. This will ensure your fixed format is actually fully honored and no user settings override your format. This also ensures month and weekday names appear in English. Without using this special locale, you may get 24-hour format even if you specify 12-hour (or visa-versa). And dates sent to a server almost always need to be in English."

Timezones

Stating the obvious one more time:

Greenwich Mean Time (GMT) is a time zone while Coordinated Universal Time (UTC) is a time standard. There is no time difference between them.

Timezones are fundamental to show the correct date/time in the final text shown to the user. The timezone value is taken from macOS and the iOS simulator inherits it, meaning that printing TimeZone.current, shows the timezone set in the macOS preferences (e.g. Europe/Berlin).

Show me some code

Note that in the following example, we use GMT (Greenwich Mean Time) and CET (Central European Time), which is GMT+1. Mind that it's best to reuse formatters since the creation is expensive.

class CustomDateFormatter {
    
    private let dateFormatter: DateFormatter = {
        let dateFormatter = DateFormatter()
        dateFormatter.dateStyle = .medium
        dateFormatter.timeStyle = .short
        return dateFormatter
    }()
    
    private let locale: Locale
    private let timeZone: TimeZone
    
    init(locale: Locale = .current, timeZone: TimeZone = .current) {
        self.locale = locale
        self.timeZone = timeZone
    }
    
    func string(from date: Date) -> String {
        dateFormatter.locale = locale
        dateFormatter.timeZone = timeZone
        return dateFormatter.string(from: date)
    }
}

let stringA = "2019-11-02T16:53:42"
let stringB = "2019-11-02T16:53:42Z"
let stringC = "2019-11-02T16:53:42-02:00"
let stringD = "2019-11-02T16:53:42.974Z"

// The ISO8601DateFormatter's extension (redacted)
// internally uses multiple formatters, each one set up with different
// options (.withInternetDateTime, .withFractionalSeconds, withFullDate, .withTime, .withColonSeparatorInTime)
// to be able to parse all the formats.
// timeZone property is set to GMT.

let dateA = ISO8601DateFormatter.date(from: stringA)!
let dateB = ISO8601DateFormatter.date(from: stringB)!
let dateC = ISO8601DateFormatter.date(from: stringC)!
let dateD = ISO8601DateFormatter.date(from: stringD)!

var dateFormatter = CustomDateFormatter(locale: Locale(identifier: "en_GB"), timeZone: TimeZone(identifier: "GMT")!)
dateFormatter.string(from: dateA) // 2 Nov 2019 at 16:53
dateFormatter.string(from: dateB) // 2 Nov 2019 at 16:53
dateFormatter.string(from: dateC) // 2 Nov 2019 at 18:53
dateFormatter.string(from: dateD) // 2 Nov 2019 at 16:53

dateFormatter = CustomDateFormatter(locale: Locale(identifier: "it_IT"), timeZone: TimeZone(identifier: "CET")!)
dateFormatter.string(from: dateA) // 2 nov 2019, 17:53
dateFormatter.string(from: dateB) // 2 nov 2019, 17:53
dateFormatter.string(from: dateC) // 2 nov 2019, 19:53
dateFormatter.string(from: dateD) // 2 nov 2019, 17:53

Using the CET timezone also for ISO8601DateFormatter, the final string produced for dateA would respectively be "15:53" when formatted with GMT and "16:53" when formatted with CET. As long as the string passed to ISO8601DateFormatter is in UTC, it's irrelevant to set the timezone on the formatter.

Apple suggests to set the timeZone property to UTC with TimeZone(secondsFromGMT: 0), but this is irrelevant if the string representing the date already includes the timezone. If your server returns a string representing a date that is not in UTC, it's probably because of one of the following 2 reasons:

1. it's not meant to be in UTC (questionable design decision indeed) and therefore the timezone of the device should be used instead;
2. the backend developers implemented it wrong and they should add the 'Z 'at the end of the string if what they intended is to have the date in UTC.

In short:

All DateFormatters should have timezone and locale set to .current and avoid handling non-UTC string if possible.

Formatting currencies € $ ¥ £

The currency symbol and the formatting of a number should be defined via a Locale, and they shouldn't be set/changed on the NumberFormatter. Don't use the user's locale (Locale.current) because it could be set to a region not supported by the app.

Let's consider the example of a user's locale to be en_US, and the app to be available only for the Italian market. We must set a locale Locale(identifier: "it_IT") on the formatter, so that:

• prices will be shown only in Euro (not American Dollar)
• the format used will be the one of the country language (for Italy, "12,34 €", not any other variation such as "€12.34")

class CurrencyFormatter {
    
    private let locale: Locale
    
    init(locale: Locale = .current) {
        self.locale = locale
    }

    func string(from decimal: Decimal,
                overriddenCurrencySymbol: String? = nil) -> String {
        let formatter = NumberFormatter()
        formatter.numberStyle = .currency
        if let currencySymbol = overriddenCurrencySymbol {
            // no point in doing this on a NumberFormatter ❌
            formatter.currencySymbol = currencySymbol
        }
        formatter.locale = locale
        return formatter.string(from: decimal as NSNumber)!
    }
}

let itCurrencyFormatter = CurrencyFormatter(locale: Locale(identifier: "it_IT"))
let usCurrencyFormatter = CurrencyFormatter(locale: Locale(identifier: "en_US"))
let price1 = itCurrencyFormatter.string(from: 12.34) // "12,34 €" ✅
let price2 = usCurrencyFormatter.string(from: 12.34) // "$12.34" ✅

let price3 = itCurrencyFormatter.string(from: 12.34, overriddenCurrencySymbol: "₿") // "12,34 ₿" ❌
let price4 = usCurrencyFormatter.string(from: 12.34, overriddenCurrencySymbol: "₿") // "₿ 12.34" ❌

In short:

All NumberFormatters should have the locale set to the one of the country targeted and no currencySymbol property overridden (it's inherited from the locale).

Languages 🇬🇧 🇮🇹 🇳🇱

Stating the obvious one more time, but there are very rare occasions that justify forcing the language in the app:

func setLanguage(_ language: String) {
    let userDefaults = UserDefaults.standard
    userDefaults.set([language], forKey: "AppleLanguages")
}

The above circumvents the Apple localization mechanism and needs an app restart, so don't do it and localize the app by the book:

• add localizations in Project -> Localizations;
• create a Localizable.strings file and tap the localize button in the inspector;
• always use NSLocalizedString() in code.

Let's consider this content of Localizable.strings (English):

"kHello" = "Hello";
"kFormatting" = "Some formatting 1. %@ 2. %d.";

and this for another language (e.g. Italian) Localizable.strings (Italian):

"kHello" = "Ciao";
"kFormatting" = "Esempio di formattazione 1) %@ 2) %d.";

Simple localization

Here's the trivial example:

let localizedString = NSLocalizedString("kHello", comment: "")

If Locale.current.languageCode is it, the value would be 'Ciao', and 'Hello' otherwise.

Formatted localization

For formatted strings, use the following:

let stringWithFormats = NSLocalizedString("kFormatting", comment: "")
String.localizedStringWithFormat(stringWithFormats, "some value", 3)

As before, if Locale.current.languageCode is it, value would be 'Esempio di formattazione 1) some value 2) 3.', and 'Some formatting 1) some value 2) 3.' otherwise.

Plurals localization

For plurals, create a Localizable.stringsdict file and tap the localize button in the inspector. Localizable.strings and Localizable.stringsdict are independent, so there are no cross-references (something that often tricked me).

Here is a sample content:

<dict>
    <key>kPlurality</key>
    <dict>
        <key>NSStringLocalizedFormatKey</key>
        <string>Interpolated string: %@, interpolated number: %d, interpolated variable: %#@COUNT@.</string>
        <key>COUNT</key>
        <dict>
            <key>NSStringFormatSpecTypeKey</key>
            <string>NSStringPluralRuleType</string>
            <key>NSStringFormatValueTypeKey</key>
            <string>d</string>
            <key>zero</key>
            <string>nothing</string>
            <key>one</key>
            <string>%d object</string>
            <key>two</key>
            <string></string>
            <key>few</key>
            <string></string>
            <key>many</key>
            <string></string>
            <key>other</key>
            <string>%d objects</string>
        </dict>
    </dict>
</dict>

Localizable.stringsdict undergo the same localization mechanism of its companion Localizable.strings. It's mandatory to only implement 'other', but an honest minimum includes 'zero', 'one', and 'other'. Given the above content, the following code should be self-explanatory:

let localizedHello = NSLocalizedString("kHello", comment: "") // from Localizable.strings
let stringWithPlurals = NSLocalizedString("kPlurality", comment: "") // from Localizable.stringsdict
String.localizedStringWithFormat(stringWithPlurals, localizedHello, 42, 1)

With the en language, the value would be 'Interpolated string: Hello, interpolated number: 42, interpolated variable: 1 object.'.

Use the scheme's option to run with a specific Application Language (it will change the current locale language and therefore also the output of the DateFormatters).

If the language we've set or the device language are not supported by the app, the system falls back to en.

References

• https://en.wikipedia.org/wiki/ISO_8601
• https://nsdateformatter.com/
• https://foragoodstrftime.com/
• https://epochconverter.com/

So... that's all folks. 🌍

Originally published on the Just Eat Engineering Blog.

Overview

Modular mobile architectures have been a hot topic over the past 2 years, counting a plethora of articles and conference talks. Almost every big company promoted and discussed modularization publicly as a way to scale big projects. At Just Eat, we jumped on the modular architecture train probably before it was mainstream and, as we'll discuss in this article, the root motivation was quite peculiar in the industry.

Over the years (2016-2019), we've completely revamped our iOS products from the ground up and learned a lot during this exciting and challenging journey. There is so much to say about the way we structured our iOS stack that it would probably deserve a series of articles, some of which have previously been posted. Here we summarize the high-level iOS architecture we crafted, covering the main aspects in a way concise enough for the reader to get a grasp of them and hopefully learn some valuable tips.

Modular Architecture

Lots of information can be found online on modular architectures. In short:

A modular architecture is a software design technique that emphasizes separating the functionality of a program into independent, interchangeable modules, such that each one contains everything necessary to execute only one aspect of the desired functionality.

Note that modular design applies to the code you own. A project with several third-party dependencies but no sensible separation for the code written by your team is not considered modular.

A modular design is more about the principle rather than the specific technology. One could achieve it in a variety of ways and with different tools. Here are some key points and examples that should inform the decision of the ifs and the hows of implementing modularization:

Business reasons

• The company requires that parts of the codebase are reused and shared across projects, products, and teams;
• The company requires multiple products to be unified into a single one.

Tech reasons

• The codebase has grown to a state where things become harder and harder to maintain and to iterate over;
• Development is slowed down due to multiple developers working on the same monolithic codebase;
• Besides reusing code, you need to port functionalities across projects/products.

Multiple teams

• The company structured teams following strategic models (e.g. Spotify model) and functional teams only work on a subset of the final product;
• Ownership of small independent modules distributed across teams enables faster iterations;
• The much smaller cognitive overhead of working on a smaller part of the whole product can vastly simplify the overall development.

Pre-existing knowledge

• Members of the team might already be familiar with specific solutions (Carthage, CocoaPods, Swift Package Manager, manual frameworks setup within Xcode). In the case of a specific familiarity with a system, it's recommended to start with it since all solutions come with pros and cons and there's not a clear winner at the time of writing.

Modularizing code (if done sensibly) is almost always a good thing: it enforces separation of concerns, keeps complexity under control, allows faster development, etc. It has to be said that it's not necessarily what one needs for small projects and its benefits become tangible only after a certain complexity threshold is crossed.

Journey to a new architecture

In 2014, Just Eat was a completely different environment from today and back then the business decided to split the tech department into separate departments: one for UK and one for the other countries. While this was done with the best intentions to allow faster evolution in the main market (UK), it quickly created a hard division between teams, services, and people. In less than 6 months, the UK and International APIs and consumer clients deeply diverged introducing country-specific logic and behaviors.

By mid-2016 the intent of "merging back" into a single global platform was internally announced and at that time it almost felt like a company acquisition. This is when we learned the importance of integrating people before technology.

The teams didn’t know each other very well and became reasonably territorial on their codebase. It didn’t help that the teams span multiple cities. It's understandable that getting to an agreement on how going back to a single, global, and unified platform took months. The options we considered spanned from rewriting the product from scratch to picking one of the two existing ones and make it global. A complete rewrite would have eventually turned out to be a big-bang release with the risk of regressions being too high; not something sensible or safe to pursue. Picking one codebase over the other would have necessarily let down one of the two teams and caused the re-implementation of some missing features present in the other codebase. At that time, the UK project was in a better shape and new features were developed for the UK market first. The international project was a bit behind due to the extra complexity of supporting multiple countries and features being too market-specific.

During that time, the company was also undergoing massive growth and with multiple functional teams having been created internally, there was an increasing need to move towards modularization. Therefore, we decided to gradually and strategically modularize parts of the mobile products and onboard them onto the other codebase in a controlled and safe way. In doing so, we took the opportunity to deeply refactor and, in the vast majority of the cases, rewrite parts in their entirety enabling new designs, better tests, higher code coverage, and - holistically - a fully Swift codebase.

We knew that the best way to refactor and clean up the code was by following a bottom-up approach. We started with the foundations to solve small and well-defined problems - such as logging, tracking, theming - enabling the team to learn to think modular. We later moved to isolating big chunks of code into functional modules to be able to onboard them into the companion codebase and ship them on a phased rollout. We soon realized we needed a solid engine to handle run-time configurations and remote feature flagging to allow switching ON and OFF features as well as entire modules. As discussed in a previous article, we developed JustTweak to achieve this goal.

At the end of the journey, the UK and the International projects would look very similar, sharing a number of customizable modules, and differing only in the orchestration layer in the apps.

The Just Eat iOS apps are far bigger and more complex than they might look at first glance. Generically speaking, merging different codebases takes orders of magnitude longer than separating them, and for us, it was a process that took over 3 years, being possible thanks to unparalleled efforts of engineers brought to work together. Over this time, the whole team learned a lot, from the basics of developing code in isolation to how to scale a complex system.

Holistic Design 🤘

The following diagram outlines the modular architecture in its entirety as it is at the time of writing this article (December 2019). We can appreciate a fair number of modules clustered by type and the different consumer apps.

Whenever possible, we took the opportunity to abstract some modules having them in a state that allows open-sourcing the code. All of our open-source modules are licensed under Apache 2 and can be found at github.com/justeat.

Apps

Due to the history of Just Eat described above, we build different apps

• per country
• per brand
• from different codebases

All the modularization work we did bottom-up brought us to a place where the apps differ only in the layer orchestrating the modules. With all the consumer-facing features been moved to the domain modules, there is very little code left in the apps.

Domain Modules

Domain modules contain features specific to an area of the product. As the diagram above shows, the sum of all those parts makes up the Just Eat apps. These modules are constantly modified and improved by our teams and updating the consumer apps to use newer versions is an explicit action. We don't particularly care about backward compatibility here since we are the sole consumers and it's common to break the public interface quite often if necessary. It might seem at first that domain modules should depend on some Core modules (e.g. APIClient) but doing so would complicate the dependency tree as we'll discuss further in the "Dependency Management" section of this article. Instead, we inject core modules' services, simply making them conformant to protocols defined in the domain module. In this way, we maintain a good abstraction and avoid tangling the dependency graph.

Core & Shared modules

The Core and Shared modules represent the foundations of our stack, things like:

• custom UI framework
• theming engine
• logging, tracking, and analytics libraries
• test utilities
• client for all the Just Eat APIs
• feature flagging and experimentation engine

and so forth. These modules - which are sometimes also made open-source - should not change frequently due to their nature. Here backward compatibility is important and we deprecate old APIs when introducing new ones. Both apps and domain modules can have shared modules as dependencies, while core modules can only be used by the apps. Updating the backbone of a system requires the propagation of the changes up in the stack (with its maintenance costs) and for this reason, we try to keep the number of shared modules very limited.

Structure of a module

As we touched on in previous articles, one of our fundamental principles is "always strive to find solutions to problems that are scalable and hide complexity as much as possible". We are almost obsessed with making things as simple as they can be.

When building a module, our root principle is:

Every module should be well tested, maintainable, readable, easily pluggable, and reasonably documented.

The order of the adjectives implies some sort of priority.

First of all, the code must be unit tested, and in the case of domain modules, UI tests are required too. Without reasonable code coverage, no code is shipped to production. This is the first step to code maintainability, where maintainable code is intended as "code that is easy to modify or extend". Readability is down to reasonable design, naming convention, coding standards, formatting, and all that jazz.

Every module exposes a Facade that is very succinct, usually no more than 200 lines long. This entry point is what makes a module easily pluggable. In our module blueprint, the bare minimum is a combination of a facade class, injected dependencies, and one or more configuration objects driving the behavior of the module (leveraging the underlying feature flagging system powered by JustTweak discussed in a previous article).

The facade should be all a developer needs to know in order to consume a module without having to look at implementation details. Just to give you an idea, here is an excerpt from the generated public interface of the Account module (not including the protocols):

public typealias PasswordManagementService = ForgottenPasswordServiceProtocol & ResetPasswordServiceProtocol
public typealias AuthenticationService = LoginServiceProtocol & SignUpServiceProtocol & PasswordManagementService & RecaptchaServiceProtocol
public typealias UserAccountService = AccountInfoServiceProtocol & ChangePasswordServiceProtocol & ForgottenPasswordServiceProtocol & AccountCreditServiceProtocol

public class AccountModule {
    public init(settings: Settings,
                authenticationService: AuthenticationService,
                userAccountService: UserAccountService,
                socialLoginServices: [SocialLoginService],
                userInfoProvider: UserInfoProvider)

    public func startLogin(on viewController: UIViewController) -> FlowCoordinator
    public func startResetPassword(on viewController: UIViewController, token: Token) -> FlowCoordinator
    public func startAccountInfo(on navigationController: UINavigationController) -> FlowCoordinator
    public func startAccountCredit(on navigationController: UINavigationController) -> FlowCoordinator
    public func loginUsingSharedWebCredentials(handler: @escaping (LoginResult) -> Void)
}

We believe code should be self-descriptive and we tend to put comments only on code that really deserves some explanation, very much embracing John Ousterhout's approach described in A Philosophy of Software Design. Documentation is mainly relegated to the README file and we treat every module as if it was an open-source project: the first thing consumers would look at is the README file, and so we make it as descriptive as possible.

Overall design

We generate all our modules using CocoaPods via $ pod lib create which creates the project with a standard template generating the Podfile, podspec, and demo app in a breeze. The podspec could specify additional dependencies (both third-party and Core modules) that the demo app's Podfile could specify core modules dependencies alongside the module itself which is treated as a development pod as per standard setup.

The backbone of the module, which is the framework itself, encompasses both business logic and UI meaning that both source and asset files are part of it. In this way, the demo apps are very much lightweight and only showcase module features that are implemented in the framework.

The following diagram should summarize it all.

Demo Apps

Every module comes with a demo app we give particular care to. Demo apps are treated as first-class citizens and the stakeholders are both engineers and product managers. They massively help to showcase the module features - especially those under development - vastly simplify collaboration across Engineering, Product, and Design, and force a good mock-based test-first approach.

Following is a SpringBoard page showing our demo apps, very useful to individually showcase all the functionalities implemented over time, some of which might not surface in the final product to all users. Some features are behind experiments, some still in development, while others might have been retired but still present in the modules.

Every demo app has a main menu to:

• access the features
• force a specific language
• toggle configuration flags via JustTweak
• customize mock data

We show the example of the Account module demo app on the right.

Internal design

It's worth noting that our root principle mentioned above does not include any reference to the internal architecture of a module and this is intentional. It's common for iOS teams in the industry to debate on which architecture to adopt across the entire codebase but the truth is that such debate aims to find an answer to a non-existing problem. With an increasing number of modules and engineers, it's fundamentally impossible to align on a single paradigm shared and agreed upon by everyone. Betting on a single architectural design would ultimately let down some engineers who would complain down the road that a different design would have played out better.

We decided to stick with the following rule of thumb:

Developers are free to use the architectural design they feel would work better for a given problem.

This approach brought us to have a variety of different designs - spanning from simple old-school MVC, to a more evolved VIPER - and we constantly learn from each other's code. What's important at the end of the day is that techniques such as inversion of control, dependency injection, and more generally the SOLID principles, are used appropriately to embrace our root principle.

Dependency Management

We rely heavily on CocoaPods since we adopted it in the early days as it felt like the best and most mature choice at the time we started modularizing our codebase. We think this still holds at the time of writing this article but we can envision a shift to SPM (Swift Package Manager) in 1-2 years time.

With a growing number of modules, comes the responsibility of managing the dependencies between them. No panacea can cure dependency hell, but one should adopt some tricks to keep the complexity of the stack under reasonable control.

Here's a summary of what worked for us:

• Always respect semantic versioning;
• Keep the dependency graph as shallow as possible. From our apps to the leaves of the graph there are no more than 2 levels;
• Use a minimal amount of shared dependencies. Be aware that every extra level with shared modules brings in higher complexity;
• Reduce the number of third-party libraries to the bare minimum. Code that's not written and owned by your team is not under your control;
• Never make modules within a group (domain, core, shared) depend on other modules of the same group;
• Automate the publishing of new versions. When a pull request gets merged into the master branch, it must also contain a version change in the podspec. Our continuous integration system will automatically validate the podspec, publish it to our private spec repository, and in just a matter of minutes the new version becomes available;
• Fix the version for dependencies in the Podfile. Whether it is a consumer app or a demo app, we want both our modules and third-party libraries not to be updated unintentionally. It's acceptable to use the optimistic operator for third-party libraries to allow automatic updates of new patch versions;
• Fix the version for third-party libraries in the modules' podspec. This guarantees that modules' behavior won't change in the event of changes in external libraries. Failing to do so would allow defining different versions in the app's Podfile, potentially causing the module to not function correctly or even to not compile;
• Do not fix the version for shared modules in the modules' podspec. In this way, we let the apps define the version in the Podfile, which is particularly useful for modules that change often, avoiding the hassle of updating the version of the shared modules in every podspec referencing it. If a new version of a shared module is not backward compatible with the module consuming it, the failure would be reported by the continuous integration system as soon as a new pull request gets raised.

A note on the Monorepo approach

When it comes to dependency management it would be unfair not to mention the opinable monorepo approach. Monorepos have been discussed quite a lot by the community to pose a remedy to dependency management (de facto ignoring it), some engineers praise them, others are quite contrary. Facebook, Google, and Uber are just some of the big companies known to have adopted this technique, but in hindsight, it's still unclear if it was the best decision for them.

In our opinion, monorepos can sometimes be a good choice.

For example, in our case, a great benefit a monorepo would give us is the ability to prepare a single pull request for both implementing a code change in a module and integrating it into the apps. This will have an even greater impact when all the Just Eat consumer apps are globalized into a single codebase.

Onwards and upwards

Modularizing the iOS product has been a long journey and the learnings were immense. All in all, it took more than 3 years, from May 2016 to October 2019, always balancing tech and product improvements. Our natural next step is unifying the apps into a single global project, migrating the international countries over to the UK project to ultimately reach the utopian state of having a single global app. All the modules have been implemented in a fairly abstract way and following a white labeling approach, allowing us to extend support to new countries and onboard acquired companies in the easiest possible way.