Journalctl : add _SYSTEMD_UNIT field into log printout

Question

Using the command:

/usr/bin/journalctl -o short -f | ncat {some-ip} {some port}

To forward journal output to some remote log tracking app.

Problem is that I'm missing the systemd unit / service name in the printout making it hard to tell which service produces what log line.

for example this is a nginx line :

Jun 25 07:51:09 localhost bash[497]: 10.23.132.98 - - [25/Jun/2014:07:51:09 +0000] "GET /page.html HTTP/1.1" 200 321 "https://{ip}" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36"

In the log there is bash[497] - the PID of the process. How can I add more data to the log ? For example the docker container name of this PID or the systemd service/unit name ?

Nimrod007 · Accepted Answer · 2014-06-26 12:34:55Z

1

In the end ive implemented this in a diffrent way

each service / systemd unit has a post execute script that uses :

/usr/bin/journalctl -u {unit name} -o short -f | sed 's/^/{unit name}/' | ncat {some-ip} {some port}

now i have the unit name in the begining of the log line ! and my log collector has unit names in the message !

example

journalctl -u mongodb.service -o short -f | sed 's/^/mongodb.service /' | ncat {some-ip} {some port}

will output :

mongodb Jun 26 09:11:35 localhost bash[1710]: 2014-06-26T09:11:35.714+0000 [rsHealthPoll] replset info mongodb-0:27017 heartbeat failed, retrying

edited Jun 26, 2014 at 12:34

answered Jun 26, 2014 at 9:14

Nimrod007

9,9518 gold badges50 silver badges71 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

user185953 Over a year ago

That is what I ended up doing. -o full and -o json are no good for humans

daurnimator · Accepted Answer · 2014-08-04 01:35:42Z

1

Don't use -o short, it removes information!

try -o json or -o verbose

answered Aug 4, 2014 at 1:35

daurnimator

4,36121 silver badges37 bronze badges

3 Comments

daurnimator Over a year ago

Most log software can ingest many formats. But try verbose then. Otherwise you could pipe it through some other program to format it as your software supports.

user185953 Over a year ago

What about humans? Monitoring multiple units with journalctl -f -u someprefix* works like charm, but how I add unit information?

M. Rubio-Roy Over a year ago

@user185953 you can use --output=with-unit

gm3dmo · Accepted Answer · 2015-06-01 07:51:32Z

1

Python can do this:

from systemd import journal

j = journal.Reader()
j.this_boot()
j.add_match(_SYSTEMD_UNIT="newnginx.service")
for entry in j:
   print('{} {}'.format(entry['_SYSTEMD_UNIT'], entry['MESSAGE']))

(python on CentOS 7)

answered Jun 1, 2015 at 7:51

gm3dmo

3456 silver badges17 bronze badges

Comments

chrisber · Accepted Answer · 2018-04-24 15:08:34Z

0

Instead of _SYSTEMD_UNIT field use the container name.

I landed here because docker stack/swarm is missing the docker-compose logs feature to log all service on a single node.

With Docker Server Version: 18.03.0-ce and docker compose version 3.6 it is possible to send all logs to journald and log them by service name.In order to identify each container, tag them with the image name. See the logging section of docker-compose.dev.yml

Docker Compose:

version: '3.6'
networks:
    skynet:
    driver: overlay
services:
    mongo:
        image: mongo:3
        networks:
            - skynet
        volumes:
            - /data/mongodb:/data/db
        logging:
            driver: "journald"
            options:
                tag: "{{.Name}}"
                labels: "com.docker.stack.namespace"

journalctl:

sudo journalctl -b -o short --all -f COM_DOCKER_STACK_NAMESPACE=skynet

Docker Command:

docker stack deploy -c docker-compose.dev.yml skynet

Will output: skynet_mongo.1.ins4s13luwiekrri3m5a8vwwl

Apr 24 14:23:08 c-wrk skynet_mongo.1.ins4s13luwiekrri3m5a8vwwl[19410]: 2018-04-24T12:23:08.212+0000 I NETWORK  [listener] connection accepted from 10.0.0.10:36518 #3 (3 connections now open)

Use full notes:

To view available labels use: docker inspect -f {{.Config.Labels}} <docker-id>

Available tags use: Customize log driver output

Config journald log driver: Docker Journald logging driver

Colorize log output: use journalctl | ccze -A

edited Apr 24, 2018 at 15:08

answered Apr 24, 2018 at 13:09

chrisber

7701 gold badge13 silver badges28 bronze badges

1 Comment

user185953 Over a year ago

Your journalctl command is about filtering the journal. This question is about displaying logs for multiple units and showing unit names with each displayed log.

Collectives™ on Stack Overflow

Journalctl : add _SYSTEMD_UNIT field into log printout

4 Answers 4

1 Comment

3 Comments

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

3 Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related