rengel
I exported a brain to JSON. When trying to read one of the JSON files using the Python json library I get an error. Some resarch showed that your json export format is not standards-conform. The error was caused by some leading bytes.
https://www.rfc-editor.org/info/rfc8259
from page 9 of the PDF download:
json1.png  According to my IDE (PyCharm) there is an additional error:

json2.png
Reinhard, TB 11.0.98.0, WIN 10

Quality is the result of attention paid to details.
Quote
mcaton
Sharing with the engineering team.  Thanks for posting.

Matt
Quote
marcusyoung
@mcaton This is still an issue. The exported JSON is not valid JSON format.
Quote
mctrexler
Perhaps that helps explain why no one seems to be able to do anything with a brain exported as JSON.   
Quote
peterB
The current JSON formatting is intentional and is used extensively by TheBrain internally.

The entire outputted chunk of text, taken as is, as a whole, is indeed not valid JSON. But each individual line is a valid JSON object. And the entire export can easily by made into valid JSON by inserting a comma after each line and wrapping the entire thing in square brackets.

This was done this way intentionally, so that it can easily be processed progressively without the need to load the entire file at once. You have two options.

1. Process each line (ending with `\n`) as a separate entity. This is what we do. Setup the reader for the file to read one line at a time and feed each time into your JSON processor.

2. Either preprocess the files and add the "missing" commas at the end of each line except the last. Add a starting `[` and an ending `]`. We don't recommend this as doing so will require the entirety of the JSON file all at once and it may be multiple megabytes large - causing scalability issues.

-Peter
Quote
marcusyoung
Hi Peter

Thanks for the information.

Of course the expectation if an export to JSON is made available is that the output is valid and doesn't require subsequent pre processing. I think there should at least be an option for which format you want when you export. 

Marcus
Quote
NickMilner
I've seen people solve the problem of processing huge JSON files that can't be loaded into memory by stripping the commas and encompassing brackets to turn it into the same format TB uses. Once done you can process the file element by element as described by Peter, above.

I don't think it's a particularly elegant solution if I'm honest, even though I suspect the majority of tools would barf at a TB JSON file - whether it was correctly formed or not - due to its size.

Another solution is to use something "jq" in streaming mode (see https://stedolan.github.io/jq/manual/#Streaming) and a bit of jq scripting to get what you need. This processes JSON files of any size without having to load them into memory first. There are streaming JSON libraries for Python, too (addressing the OP.)

My feeling is that it's better to create a standards-compliant file and let people handle it their own way, rather than create something that can't be read by any JSON parser and forces a particular solution, even if your intent is noble. 😉

/2c
Quote

Newsletter Signup  Newsletter        Visit TheBrain Blog   Blog       Follow us on Twitter   Twitter       Like Us on Facebook   Facebook         Watch Us on Youtube  YouTube       

TheBrain Mind Map & Mindmapping Software     Download TheBrain Mind Mapping Software