Python conversion from JSON to JSONL

The jsonlines package is made exactly for your use case:

import jsonlines

items = [
    {'a': 1, 'b': 2},
    {'a', 123, 'b': 456},
]
with jsonlines.open('output.jsonl', 'w') as writer:
    writer.write_all(items)

(Yes, I wrote it years after you posted your original question.)

A simple way to do this is with the jq command in your terminal.

To install jq on Debian and derivatives:

sudo apt-get install jq

CentOS and RHEL users should run:

sudo yum -y install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum install jq -y

Basic usage:

jq -c '.[]' some_json.json >> output.jsonl

If you need to handle with huge files, I strongly recommend to use the --stream flag. This will make jq parse your JSON content in streaming mode.

jq -c --stream '.[]' some_json.json >> output.json

But, if you need to do this operation into a Python file, you can use bigjson, a useful library that parses the JSON in streaming mode:

pip3 install bigjson

To read a huge JSON file (in my case, it was 40 GB):

import bigjson

# Reads JSON file in streaming mode
with open('input_file.json', 'rb') as f:
    json_data = bigjson.load(f)

    # Open output file
    with open('output_file.jsonl', 'w') as outfile:
        # Iterates over input json
        for data in json_data:
            # Converts json to a Python dict
            dict_data = data.to_python()

            # Saves the output to output file
            outfile.write(json.dumps(dict_data)+"\n")

If you want, try to parallelize this code aiming to improve performance. Post the result here :)

Documentation and source code: bigjson

Note that a JSONL file is a compacted JSON file. You may need to pass separators without spaces:

with open(output_file_jsonl, 'a', encoding ='utf8') as json_file:
    for elem in rs:
        json_file.write(json.dumps(dict(elem), separators=(',', ':'), cls=DateTimeEncoder))
        json_file.write('\n')

This is an edit to this answer which takes into account the possibility of special symbols or using a different alphabet in the JSONL file. For example, I use Cyrillic and without the encoding and ensure_ascii parameters edited, I get really ugly results. I thought it could be useful:

with open('output.jsonl', 'w', encoding='utf8') as outfile:
    for entry in dataset_donut:
        json.dump(entry, outfile, ensure_ascii=False)
        outfile.write('\n')

You might also get this done with RegEx search and replace, for example in VSCode, if you switch on RegEx and search for \n:

and replace all by nothing. Then save the changed json file as "myfile.jsonl".

That works if you have only one JSON in your file (which should be the default). If you have a list of JSONs, like in your example, you can still search and replace with RegEx, take the negative lookahead "?!", see Stack Overflow Find 'word' not followed by a certain character with \n(?!\s*\{) so that you also skip the spaces after a linebreak:

And there you go:

Clean the rest of the unneeded characters as you showed it yourself, but take RegEx for it, and you could also do this RegEx replacement automatically with Python with the re package instead of doing this in VSCode by hand.

• Replace ^\[ with nothing to get rid of the first bracket "[".
• Replace \]$ with nothing to get rid of the last bracket "]".
• Replace \,$ with nothing to get rid of the trailing commas ",".
• Replace ^\s* with nothing to get rid of spaces at the beginning of a line.

Out:

{u'index': 1,  u'no': 'A',  u'met': u'1043205'}
{u'index': 2,  u'no': 'B',  u'met': u'000031043206'}
{u'index': 3,  u'no': 'C',  u'met': u'0031043207'}]

If you don't want a library, it's easy enough to do using JSON directly.

Source

[
    {"index": 1,"no": "A","met": "1043205"},
    {"index": 2,"no": "B","met": "000031043206"},
    {"index": 3,"no": "C","met": "0031043207"}
]

Code

import json

with open("test.json", 'r') as infile:
    data = json.load(infile)
    if len(data) > 0:
        print(json.dumps([t for t in data[0]]))
        for record in data:
            print(json.dumps([v for (k,v) in record.items()]))

Result

["index", "no", "met"]
[1, "A", "1043205"]
[2, "B", "000031043206"]
[3, "C", "0031043207"]