Problem When Using Datasets to Open JSONL

Question

Problem When Using Datasets to Open JSONL

I am trying to open a JSONL format file using the datasets library. Here is my code:

from datasets import load_dataset

path = "./testdata.jsonl"
dataset = load_dataset('json', data_files=path, split='train')

The contents of testdata.jsonl are organized as follows (just for testing):

{"src":"hello","term":{"a":"aa"}}
{"src":"hi","term":{"b":"bb"}}

When I use the code above to load the dataset and attempt to print the second item, like this:

print(dataset[1])

I get the following output:

{'src': 'hi', 'term': {'a': None, 'b': 'bb'}}

Instead of the expected output:

{'src': 'hi', 'term': {'b': 'bb'}}

How can I obtain the second format of the dataset? Is it possible that I simply forgot to include a parameter?

Huzaifa Rashid · Accepted Answer · 2025-06-28 19:19:48Z

1

You are facing this issue because the datasets library (by Hugging Face) tries to automatically infer a consistent schema across all your JSONL entries.

In your case, it sees that the term field has different keys in different rows (a in the first, b in the second), so it merges them and fills missing keys with None.

answered Jun 28 at 19:19

Huzaifa Rashid

215 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

bluebingoSu Jun 29 at 18:46

Thank you for your insightful comment; it helped me understand why I was getting {'a': None, 'b': 'bb'} instead of {'b': 'bb'}. I'm curious if there are any techniques to prevent the merge from including None values, achieving the second format directly. I have attempted post-processing methods, such as removing None elements from term, but I find it cumbersome to map the datasets each time I open a JSONL file with JSON elements. Any suggestions for a more streamlined approach would be greatly appreciated. Thanks for your help!

Huzaifa Rashid Jun 30 at 16:40

Glad it helped! The issue happens because when merging, keys with None values are still included unless explicitly filtered out. One way around it is to filter out keys with None values before merging or use a helper function that does that automatically. I've run into the same problem when working with JSONL files—cleaning up every time can get tedious. Let me know if you want a small utility snippet to handle that cleanly.

bluebingoSu Jul 13 at 2:48

I apologize for my late reply! This issue has already been resolved by the library developer. You can find the relevant post here: link. Thank you very much for your help!

Collectives™ on Stack Overflow

Problem When Using Datasets to Open JSONL