Table of Contents

Introduction

When running a task orchestrator like Nomad or Kubernetes, there’s usually a bunch of different instances ( containers, micro-VMs, jails, etc. ) running, more or less ephemerally, across a fleet of servers. By default all logs would be local to the nodes actually running the stuff we want to run, making it burdensome to debug, correlate events, alert, etc., especially if the node crashes, hence why it’s a well established practice to collect all logs they emit to a central location, where all of those actions happen.

One of the most popular log management stacks is the so-called ELK ( ElasticSearch for log indexing, Logstash for parsing/transforming them, and Kibana for visualisation, with either Filebeat or Fluentd / Fluent bit for log collection), which has a few drawbacks - most notably heavy resource consumption and licence changes, the latter leading to numerous forks, which will probably result in some chaos/incompatibilities in the future.

A recent-ish contender in that space is Grafana Labs' Loki. It’s a lightweight Prometheus-inspired tool, which can be run as a bunch of microservices for better and easier scale-out, or in monolithic all-in-one mode. In contrast to ElasticSearch, it only indexes labels ( which are user defined ), the logs themselves ( chunks ) are stored as-is, separately. That makes it more flexible and much cheaper with regards to storage and compute.

There’s a variety of storage options - Cassandra (index and chunk), GCS/S3 (chunk), BigTable (chunk), DynamoDB(chunk) and the monolithic-only local storage ( which uses BoltDB internally for indexing). The last one is great for getting started/testing/non-huge projects, and the lack of redundancy ( since it writes to the local filesystem) can be offset by the storage provider ( e.g. a CSI plugin, GlusterFS, DRBD, or good old NFS). Visualisation from Loki is done with Grafana ( not really surprising since they’re both made by the same company), via the LogQL PromQL-inspired query language.

Having been using Loki for a few months in production, I’m really like it and hence this article on logs and using it on and with Nomad.

There are two main ways to get logs from Nomad tasks into Loki ( one of which only works for Docker tasks), and I’ll discuss the pros and cons of each one later on.

How logs on Nomad work

By default Nomad writes task logs to the alloc/logs folder shared by all tasks in an allocation, which is stored locally on the client node running it. That makes accessing logs slightly burdensome and look like the following:

# get all current allocations for the job
$ nomad job status random-logger-example 
ID            = random-logger-example
Name          = random-logger-example
Submit Date   = 2021-04-27T23:28:29+02:00
Type          = service
Priority      = 50
Datacenters   = dc1
Namespace     = default
Status        = running
Periodic      = false
Parameterized = false

Summary
Task Group  Queued  Starting  Running  Failed  Complete  Lost
random      0       0         1        0       0         0

Latest Deployment
ID          = beffde05
Status      = running
Description = Deployment is running

Deployed
Task Group  Desired  Placed  Healthy  Unhealthy  Progress Deadline
random      1        1       0        0          2021-04-27T23:38:29+02:00

Allocations
ID        Node ID   Task Group  Version  Desired  Status   Created  Modified
529b074e  a394f493  random      0        run      running  8s ago   3s ago

# get the logs for an allocation
$ nomad alloc logs 529b074e

As soon as there’s more than one allocation, you have to get the logs for each one by one; if you want to see the logs of an old allocation ( assuming it hasn’t been garbage collected already ), you have to run nomad job status with the -all-allocs flag to include non-current versions.

Specific log options for the task driver can be configured on multiple levels - per task, per client, or in certain cases, most notably Docker, per task driver.

On the task level one can send all logs to a central syslog server (easily adaptable to fluentd, Splunk, etc. ) like so:

task "syslog" {
      driver = "docker"
      config {
        logging {
          type = "syslog"
          config = {
            syslog-address         = "tcp://my-syslog-server:10514"
            tag                    = "${NOMAD_TASK_NAME}"
          }
        }
      }
}

This is perfectly fine, except that you have to do it on each and every task, which can quickly become burdensome, especially if your logging configuration changes one day, and you need to be on Docker Engine 20.10+, or use a logging driver that supports writing the logs locally as well as to the remote endpoint, otherwise docker logs and nomad alloc logs are unusable because the logs get sent directly. To facilitate that, there is use the new Docker driver-wide logging configuration, but then you’re limited only to Docker logging drivers, and, the bigger issue in my opinion, you’re lacking context (unless the stuff you run detects and specifically logs that context).

The, in my humble opinion, better option, is to have something that runs automatically on all client nodes, via a system job, and collects all logs, like what everyone does with their Kubernetes clusters via Promtail, Fluend/Fluentbit, Datadog agent, Splunk Connect, etc. The biggest challenges with that are:

  1. context - it’s good to know everything about where the logs are from - which task, namespace, client node, etc.
  2. discovery - how to find all new allocations to ship the logs of?

Recently, there have been some great news concerning those two challenges in regards to Docker-based tasks - this PR on Nomad that adds extra Docker labels with job, task, task group, node, namespace names, this PR that enables default logging options for the Docker driver on the Nomad side.

I’ll describe how to collect and ship logs from Nomad, and then briefly go over how to run Loki, a lightweight log aggregation tool by Grafana Labs, on Nomad.

Logs collection

Nomad makes each allocation’s logs available through the API, so ideally our logging agent should be able to connect to it directly. The only mainstream one that does that at the time of writing is:

Filebeat

Filebeat, Elastic’s log collection agent, has an auto-discovery module for Nomad, experimentally via this PR which hasn’t been released in a stable version yet. If one wants to use ElasticSearch, or any of the compatible alternatives or companions ( Logstash, Kafka), that’s great. In theory one can configure Filebeat to output to a file, and then parse that file with another logging agent ( like Vector or Promtail) and ship the logs somewhere else, but IMHO that seems too much hassle for little benefit.

Promtail

Grafana provide a log collecting companion to Loki in the form of Promtail, which is lightweight and easy to configure, but much more limited compared to filebeat, fluent* or Vector in terms of compatible sources. The biggest downside is that for Docker logs it can’t use Docker labels for extra context, and having only the container name and ID is often insufficient.

Vector

Vector is a very fast and lightweight observability agent that can collect logs and metrics, transform and ship them to a variety of backends by Timber, a logging SaaS. They were recently acquired by Datadog, who also have an agent and logging backend, so they obviously saw some value in them. And, very importantly for this story, Vector’s Docker logs source can use Docker labels to enrich the logs' context, and there’s a Loki sink ( backend ). So, let’s see about deploying Vector as a system job to collect logs from Docker.

First, we need to allow Vector to access the Docker daemon and configure Nomad to add extra metadata; the easiest and cleanest way to do that is to pass the Docker socket as a host volume, which need to be declared at the host (Nomad client) level, and then mounted in the Vector job. We can do that read-only, and we can use ACLs to limit who can mount it to avoid allowing every job to mount it and do whatever.


plugin "docker" {
  config {  
    # extra Docker labels to be set by Nomad on each Docker container with the appropriate value
    extra_labels = ["job_name", "task_group_name", "task_name", "namespace", "node_name"]
  }
}

client {
  host_volume "docker-sock-ro" {
    path = "/var/run/docker.sock"
    read_only = true
  }
}

host_volume "docker-sock-ro" {
  policy = "read" # read = read-only, write = read/write, deny to deny
}

Second, actually mount the newly available host volume inside the Vector task:



group "vector" {
    ...
    volume "docker-sock" {
      type = "host"
      source = "docker-sock"
      read_only = true
    }
    task "vector" {
      ...
      volume_mount {
        volume = "docker-sock"
        destination = "/var/run/docker.sock"
        read_only = true
      }
    }
}

Third, tell Vector to collect Docker’s logs and send them to Loki, enriching them with the metadata extracted from the Docker labels:


          data_dir = "alloc/data/vector/"
          [sources.logs]
            type = "docker_logs"
          [sinks.out]
            type = "console"
            inputs = [ "logs" ]
            encoding.codec = "json"
          [sinks.loki]
            type = "loki" 
            inputs = ["logs"] 
            endpoint = "http://loki.example"
            encoding.codec = "json" 
            healthcheck.enabled = true 
            # since . is used by Vector to denote a parent-child relationship, and Nomad's Docker labels contain ".",
            # we need to escape them twice, once for TOML, once for Vector
            labels.job = "{{ label.com\\.hashicorp\\.nomad\\.job_name }}"
            labels.task = "{{ label.com\\.hashicorp\\.nomad\\.task_name }}"
            labels.group = "{{ label.com\\.hashicorp\\.nomad\\.task_group_name }}"
            labels.namespace = "{{ label.com\\.hashicorp\\.nomad\\.namespace }}"
            labels.node = "{{ label.com\\.hashicorp\\.nomad\\.node_name }}"
            # remove fields that have been converted to labels to avoid having them twice
            remove_label_fields = true

And that’s it.

Full example Vector job file with all those components and dynamic sourcing of the Loki address from Consul:


job "vector" {
  datacenters = ["dc1"]
  # system job, runs on all nodes
  type = "system" 
  update {
    min_healthy_time = "10s"
    healthy_deadline = "5m"
    progress_deadline = "10m"
    auto_revert = true
  }
  group "vector" {
    count = 1
    restart {
      attempts = 3
      interval = "10m"
      delay = "30s"
      mode = "fail"
    }
    network {
      port "api" {
        to = 8686
      }
    }
    # docker socket volume
    volume "docker-sock" {
      type = "host"
      source = "docker-sock"
      read_only = true
    }
    ephemeral_disk {
      size    = 500
      sticky  = true
    }
    task "vector" {
      driver = "docker"
      config {
        image = "timberio/vector:0.14.X-alpine"
        ports = ["api"]
      }
      # docker socket volume mount
      volume_mount {
        volume = "docker-sock"
        destination = "/var/run/docker.sock"
        read_only = true
      }
      # Vector won't start unless the sinks(backends) configured are healthy
      env {
        VECTOR_CONFIG = "local/vector.toml"
        VECTOR_REQUIRE_HEALTHY = "true"
      }
      # resource limits are a good idea because you don't want your log collection to consume all resources available 
      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
      }
      # template with Vector's configuration
      template {
        destination = "local/vector.toml"
        change_mode   = "signal"
        change_signal = "SIGHUP"
        # overriding the delimiters to [[ ]] to avoid conflicts with Vector's native templating, which also uses {{ }}
        left_delimiter = "[["
        right_delimiter = "]]"
        data=<<EOH
          data_dir = "alloc/data/vector/"
          [api]
            enabled = true
            address = "0.0.0.0:8686"
            playground = true
          [sources.logs]
            type = "docker_logs"
          [sinks.out]
            type = "console"
            inputs = [ "logs" ]
            encoding.codec = "json"
          [sinks.loki]
            type = "loki" 
            inputs = ["logs"] 
            endpoint = "http://[[ range service "loki" ]][[ .Address ]]:[[ .Port ]][[ end ]]" 
            encoding.codec = "json" 
            healthcheck.enabled = true 
            # since . is used by Vector to denote a parent-child relationship, and Nomad's Docker labels contain ".",
            # we need to escape them twice, once for TOML, once for Vector
            labels.job = "{{ label.com\\.hashicorp\\.nomad\\.job_name }}"
            labels.task = "{{ label.com\\.hashicorp\\.nomad\\.task_name }}"
            labels.group = "{{ label.com\\.hashicorp\\.nomad\\.task_group_name }}"
            labels.namespace = "{{ label.com\\.hashicorp\\.nomad\\.namespace }}"
            labels.node = "{{ label.com\\.hashicorp\\.nomad\\.node_name }}"
            # remove fields that have been converted to labels to avoid having the field twice
            remove_label_fields = true
        EOH
      }
      service {
        check {
          port     = "api"
          type     = "http"
          path     = "/health"
          interval = "30s"
          timeout  = "5s"
        }
      }
      kill_timeout = "30s"
    }
  }
}

Overall, that’s pretty a pretty good solution for the problem, with a lightweight and fast logging agent, some context, and everything is configured from within Nomad; however, that only works for Docker tasks and requires some (light) configuration on each client node.

nomad_follower (+Promtail)

There’s a project called nomad_follower, which uses Nomad’s API to tail allocation logs. It requires no outside (of the Nomad system job running it) configuration, and compiles all the logs with metadata and context in a file which can then be scraped by a logging agent and sent to whatever logging backend you have. It’s a bit more obscure compared to Vector or Filebeat, and debugging it might require diving in the code ( speaking from experience, it didn’t accept my logfmt formatted logs because it was looking at a timestamp starting in the first 4 characters of each line (used to deal with multi-line logs), but mine started after 5.. so i had to fork and patch it). Nonetheless, coupled with a logging agent it’s a fairly decent solution that works for all task types.

Configuration is done via env variables ( like the file where to write all logs, the Nomad/Consul service tag to filter on, etc.), including Nomad API access/credentials. The logs are stored in JSON, with JSON-formatted logs inside the data field, non-JSON (like logfmt) logs are in the message field, and there’s a bunch of metadata alongside them:

  • alloc_id
  • job_name
  • job_meta
  • node_name
  • service_name
  • service_tags
  • task_meta
  • task_name

  group "log-shipping" {
    count = 1
    restart {
      attempts = 2
      interval = "30m"
      delay    = "15s"
      mode     = "fail"
    }
    ephemeral_disk {
      size = 300
    }
    task "nomad-follower" {
      driver = "docker"
      config {
        image = "sofixa/nomad_follower:latest"
      }
      env {
        VERBOSE    = 4
        LOG_TAG    = "logging"
        LOG_FILE   = "${NOMAD_ALLOC_DIR}/nomad-logs.log"
        # this is the IP of the docker0 interface
        # and Nomad has been explicitly told to listen on it so that Docker tasks can communicate with the API
        NOMAD_ADDR = "http://172.17.0.1:4646" 
        # Nomad ACL token, could be sourced via template from Vault
        NOMAD_TOKEN = "xxxx" 
      }
      # resource limits are a good idea because you don't want your log collection to consume all resources available 
      resources {
        cpu    = 100
        memory = 512
      }
    }
  }

That will result in all allocations running on the same node as an allocation from this task with the logging tag to get their logs collected and stored in ${NOMAD_ALLOC_DIR}/nomad-logs.log. Now all we need is to get something to read, parse and send those logs; considering I’d like to use Loki for storage, Promtail seems like the best option, but of course any of the alternatives could do the job just as well.

Promtail’s configuration is split into scrape (collection), pipeline that parses/transforms/extracts labels, and generic ( Loki adress, ports for healthcheck, etc.). It has a few ways to scrape logs, the one we need in this case is the static, which tails a file; and we also need to parse the various fields ( via a json pipeline stage) and mark some as labels ( so they’re indexed and we can search by them).

To scrape and parse the logs pre-collected by nomad_follower, we need a configuration similar to this one:


server:
  # port for the healthcheck
  http_listen_port: 3000
  grpc_listen_port: 0
positions:
  filename: ${NOMAD_ALLOC_DIR}/positions.yaml
client:
  url: http://loki.example/loki/api/v1/push
scrape_configs:
- job_name: local
  static_configs:
  - targets:
      - localhost
    labels:
      job: nomad
      __path__: "${NOMAD_ALLOC_DIR}/nomad-logs.log"
  pipeline_stages:
    # extract the fields from the JSON logs
    - json:
        expressions:
          alloc_id: alloc_id
          job_name: job_name
          job_meta: job_meta
          node_name: node_name
          service_name: service_name
          service_tags: service_tags
          task_meta: task_meta
          task_name: task_name
          message: message
          data: data
    # the following fields are used as labels and are indexed:
    - labels:
        job_name:
        task_name:
        service_name:
        node_name:
        service_tags:
    # an example regex to extract a field called time from within message( which is for non-JSON formatted logs,
    # so the assumption is that they're in the logfmt format,
    # and a field time= is present with a timestamp in, which is the actual timestamp of the log)
    - regex:
        expression: ".*time=\\\"(?P<timestamp>\\S*)\\\"[ ]"
        source: "message"
    - timestamp:
        source: timestamp
        format: RFC3339

A full example system job file, with nomad_follower and Promtail, with a template to dynamically source Loki’s address from within Consul:


job "log-shipping" {
  datacenters = ["dc1"]
  type = "system"
  namespace = "logs"
  update {
    max_parallel      = 1
    min_healthy_time  = "10s"
    healthy_deadline  = "3m"
    progress_deadline = "10m"
    auto_revert       = false
  }
  group "log-shipping" {
    count = 1
    network {
      port "promtail-healthcheck" {
        to = 3000
      }
    }
    restart {
      attempts = 2
      interval = "30m"
      delay    = "15s"
      mode     = "fail"
    }
    ephemeral_disk {
      size = 300
    }
    task "nomad-forwarder" {
      driver = "docker"
      env {
        VERBOSE    = 4
        LOG_TAG    = "logging"
        LOG_FILE   = "${NOMAD_ALLOC_DIR}/nomad-logs.log"
        # this is the IP of the docker0 interface
        # and Nomad has been explicitly told to listen on it so that Docker tasks can communicate with the API
        NOMAD_ADDR = "http://172.17.0.1:4646"
        # Nomad ACL token, could be sourced via template from Vault
        NOMAD_TOKEN = "xxxx"
      }
      config {
        image = "sofixa/nomad_follower:latest"
      }
      # resource limits are a good idea because you don't want your log collection to consume all resources available
      resources {
        cpu    = 100
        memory = 512
      }
    }
    task "promtail" {
      driver = "docker"
      config {
        image = "grafana/promtail:2.2.1"
        args = [
          "-config.file",
          "local/config.yaml",
          "-print-config-stderr",
        ]
        ports = ["promtail-healthcheck"]
      }
      template {
        data = <<EOH
server:
  http_listen_port: 3000
  grpc_listen_port: 0

positions:
  filename: ${NOMAD_ALLOC_DIR}/positions.yaml

client:
  url: http://loki.example/loki/api/v1/push
scrape_configs:
- job_name: local
  static_configs:
  - targets:
      - localhost
    labels:
      job: nomad
      __path__: "${NOMAD_ALLOC_DIR}/nomad-logs.log"
  pipeline_stages:
    # extract the fields from the JSON logs
    - json:
        expressions:
          alloc_id: alloc_id
          job_name: job_name
          job_meta: job_meta
          node_name: node_name
          service_name: service_name
          service_tags: service_tags
          task_meta: task_meta
          task_name: task_name
          message: message
          data: data
    # the following fields are used as labels and are indexed:
    - labels:
        job_name:
        task_name:
        service_name:
        node_name:
        service_tags:
    # use a regex to extract a field called time from within message( which is for non-JSON formatted logs,
    # so the assumption is that they're in the logfmt format,
    # and a field time= is present with a timestamp in, which is the actual timestamp of the log)
    - regex:
        expression: ".*time=\\\"(?P<timestamp>\\S*)\\\"[ ]"
        source: "message"
    - timestamp:
        source: timestamp
        format: RFC3339
EOH
        destination = "local/config.yaml"
      }
      # resource limits are a good idea because you don't want your log collection to consume all resources available
      resources {
        cpu    = 500
        memory = 512
      }
      service {
        name = "promtail"
        port = "promtail-healthcheck"
        check {
          type     = "http"
          path     = "/ready"
          interval = "10s"
          timeout  = "2s"
        }
      }
    }
  }
}

Running Loki

Note: I personally consider it somewhat of an anti-pattern to run your log aggregation system on the same cluster as the one it’s aggregating logs from. If the cluster explodes, you can’t really access the logs to debug what happened and why. For smaller use cases, or if critical failure isn’t probable, it’s perfectly fine and will probably never cause any issues; past a certain point though, I’d recommend splitting it ( and other similarly scoped tools like monitoring) into a separate management cluster, or using the hosted version, which includes a decent free tier (50GB and 14 days retention when it comes to logs).

Like I mentioned last time, even though one can include YAML configuration from Consul in Nomad jobs via the template stanza, I prefer to have as much as possible in the Nomad file directly for better versioning and rollbackability ( YAML stored in Consul evolves independently of the Nomad job’s lifecycle, so rollbacking the latter won’t do anything about the former ).

To run a basic, monolithic Loki service with local storage and unlimited retention, a host volume for said storage, Traefik as a reverse proxy (with TLS and a basic auth middleware attached, since Loki doesn’t do auth itself) and Loki’s YAML configuration file embedded, you need something along these lines:

job "loki" {
  datacenters = ["dc1"]
  type        = "service"
  update {
    max_parallel      = 1
    health_check      = "checks"
    min_healthy_time  = "10s"
    healthy_deadline  = "3m"
    progress_deadline = "5m"
  }
  group "loki" {
    count = 1
    restart {
      attempts = 3
      interval = "5m"
      delay    = "25s"
      mode     = "delay"
    }
    network {
      port "loki" {
        to = 3100
      }
    }
    volume "loki" {
      type      = "host"
      read_only = false
      source    = "loki"
    }
    task "loki" {
      driver = "docker"
      config {
        image = "grafana/loki:2.2.1"
        args = [
          "-config.file",
          "local/loki/local-config.yaml",
        ]
        ports = ["loki"]
      }
      volume_mount {
        volume      = "loki"
        destination = "/loki"
        read_only   = false
      }
      template {
        data = <<EOH
auth_enabled: false
server:
  http_listen_port: 3100
ingester:
  lifecycler:
    address: 127.0.0.1
    ring:
      kvstore:
        store: inmemory
      replication_factor: 1
    final_sleep: 0s
  # Any chunk not receiving new logs in this time will be flushed
  chunk_idle_period: 1h       
  # All chunks will be flushed when they hit this age, default is 1h
  max_chunk_age: 1h           
  # Loki will attempt to build chunks up to 1.5MB, flushing if chunk_idle_period or max_chunk_age is reached first
  chunk_target_size: 1048576  
  # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
  chunk_retain_period: 30s    
  max_transfer_retries: 0     # Chunk transfers disabled
schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
storage_config:
  boltdb_shipper:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem
limits_config:
  reject_old_samples: true
  reject_old_samples_max_age: 168h
chunk_store_config:
  max_look_back_period: 0s
table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
EOH
        destination = "local/loki/local-config.yaml"
      }
      resources {
        cpu    = 512
        memory = 256
      }
      service {
        name = "loki"
        port = "loki"
        check {
          name     = "Loki healthcheck"
          port     = "loki"
          type     = "http"
          path     = "/ready"
          interval = "20s"
          timeout  = "5s"
          check_restart {
            limit           = 3
            grace           = "60s"
            ignore_warnings = false
          }
        }
        tags = [
          "traefik.enable=true",
          "traefik.http.routers.loki.tls=true",
          # the middleware has to be declared somewhere else, we only attach it here
          "traefik.http.routers.loki.middlewares=loki-basicauth@file", 
        ]
      }
    }
  }
}

Conclusion

So, which logging agent to use with Nomad ? As always, it depends. If you have ElasticSearch or any of the compatible alternatives, Filebeat is probably the best option due to the native support. If you only run Docker-based tasks, Vector collecting Docker Daemon’s logs (with Docker labels for context) is a pretty decent option. If neither, or you want more context ( like custom metadata) than what is available with Vector, nomad_follower is for you.

Discuss

Hacker News

Reddit

Twitter

Telegram