Append arrays to an existing JSON file with Jackson

Jackson is the most powerful library for JSON serialization and deserialization. It can do anything literally with JSON. But at the same time, it is a bit frustrating. That’s because Jackson has a complex workflow. This is necessary not the library fault. It’s just how JVM languages especially Java works. Anyhow, in this article, I cover how to append arrays to an existing JSON file with Jackson.

Let say we want to extract a million record from a database and convert it to a JSON file. If the data volume was small, we could just query the table, get data as Array<?> and then use this code to convert it to JSON.

The problem here is the volume of data is too large. So neither database is capable of returning a million records, nor the application can handle holding the amount of data in heap. So the only way to get it through is by reading the data page by page. But wait a second how to convert them to JSON?

If we modify the previous approach we can append the data to the list but soon we get OutOfMemoryError. Not to mention that serializing and deserializing slows down the process significantly.

A better solution is to read the data page by page and append it to the existing JSON file. That way the database only gets a single page at the time, so JVM keeps a very small array in heap.

So we need to configure Jackson in such a way that we can keep appending array to it. So that, we can do something like one in below.

For new file,

For existing file,

The first line creates or opens an existing file. Next, it reads the content as JSON array and then in appends a list to the end of the file.

For example, if the file is empty the line 2 creates an empty opening array [. Then the line 6 appends the content to the empty array,

And finally the line 11 closes the array. So we will have something like below,

If the file is not empty the same process will be repeated but with the existing content.

That’s all for this article, hope you have enjoyed it. To learn how to read or write to/from a JSON file with Jackson read my previous article available at the link below,
https://www.geekyhacker.com/2019/06/08/interacting-with-a-json-file-using-jackson-library/