mctrexler Show full post »
mctrexler
Just to report that the additional memory, assuming I'm actually accessing it per my prior post, is not having any effect on the mini-freezes.

I've also played a bit more with the Wifi issue and am not really seeing the correlation I thought I saw the first time.  So that may be a dead end as well. 

Mark
Quote
enkidu
hello again

you're on windows, right ? Are you using TheBrain + Java 32 bit or 64 bit ?

also : is your disk defragmented >> have you noticed a speed difference in TheBrain before / after defrag ??

cheers
enkidu


Quote
mctrexler
Windows 7, Java 7.0.55  32 bit.  Brain 8.0.1.1.

I've actually never defragmented the disk - it's less than 50% full, and only 13% fragmented.  Any reason to believe defragmenting would help with those numbers?  Probably couldn't hurt. 

Mark
Quote
Brigitte
You may want to give it a try to defragment. It certainly cannot hurt.

Brigitte
TheBrain Technologies
Quote
bmac
This is not a defragmentation issue.  I have a 256GB Toshiba SSD as my main drive and SSDs do not become fragmented.
Bill
customized Desktop PC (Haswell i7 4770 3.4 gHz with 32 GB RAM), Windows 10 Pro 64-bit, 256 GB SSD,  and a Microsoft Surface Pro 6, Windows 10, 512 GB SSD, 16 GB ram, i5
Quote
Spacenexus
Bill, that's dissapointing to hear. I've been thinking of upgrading to a 1TB SSD in an effort to alleviate the brain freezes on my megabrain but am concerned to hear that it still arises with an SSD. I am coming to the reluctant conclusion that there may be a combination of software limitations with the size and integrity of the brain data set combined with inadequate support to solve the issue in a secure manner. The last thing i would want to do is to hand over an integrated data set from multiple sources many of whom are under NDA for confidentiality just to address a software problem. 

Despite professional licensing formats, somewhere along the line this gap in addressing a performance / security issue has arisen and looks unlikely to be solved.

It is either an endemic problem or I also suspect it is related to the critical data failure Zenrain suffered 2-3 years ago which corrupted his entire dataset seemingly beyond repair. The developers did take his data and reassemble his brain 'manually' it seems. The only solution acceptable to both parties that is secure and scales is to build a user tool to self-diagnose and repair the dataset. I welcome any thoughts from Matt or the team on this serious issue.

best

Jim

@181218:
TB8022 32bit
Java 32bit Version 8 Update 141

Firefox, Office 2013 Pro Plus 32bit
64bit Win10Pro
64bit Primary Laptop, 8GB RAM, Intel Core i7
64bit Secondary Laptop, 64GB RAM, Intel Xeon E3
Brain user since zygote
Quote
zenrain
The corrupted brain was only 5-6,000 thoughts. I don't recall having mini-freezes, but it's an order of magnitude lower than 100k thoughts, so that doesn't mean much.
Did everyone experiencing this issue use 6x? The reason I ask is the issue was in 6.0.5.4, so if someone is experiencing freezes that has only used 7x or higher, it's not the same problem.

The corruption issue manifested originally in notes disappearing, and being transposed on other thoughts. I seem to remember attachments switching too, but this was much rarer (I'm a very heavy notes user).
TheBrain did work on my database for quite some time, and they sent back a fixed version and a list of thoughts with notes that couldn't be reconciled. They did introduce a tool to resolve this issue, it's enabled under utilities by typing *support*. It's called Remove Redundant notes. I believe Overhaul the Database may have also been introduced as part of a solution also.

I used the database for a while after this, but kept finding other notes missing, or transferred. Other problems cropped up, searches not finding thoughts or attachments, thoughts not copying correctly or causing the copy function to outright fail, and I ended up copying pertinent thoughts to a new database. Eventually I had enough problems in that database to give up on it (one item of interest was that extended searches were attributing results to thought links that existed on completely unrelated thoughts). I made an assumption that copying some thoughts into the brain introduced the issue again. 
I've since created a whole new brain (with no copied data) and have had no problems. It's still under 2,000 thoughts so early days.
I don't know if the other issues I experienced were a result of this corruption, I attributed the problems to it, but it's just a guess.
I'm giving this information to provide you as much information as I can about the issue, and what else I experienced. 
I'm under the impression that the corruption is not easily fixed because of all the other issues that I experienced, even after the fix. I would attribute the difficulty to the database complexity and perhaps synchronization exacerbating the issue. This however is conjecture.

As someone who works for a software company and am involved in identifying and trending issues, unfortunately I can corroborate that if QA and Engineering can't replicate the issue in house it's practically impossible to resolve. This is why they request the data. Unless they can replicate, go into the data and find the type and source of the corruption there is not much they can do. This doesn't help if you have a sensitive database that can't be shared. My only suggestion here is it possible to delete the NDA thoughts (in a duplicated Brain of course), and if the freezes send that data in? Yeah, that's a lot of work, I'm sorry, I just don't have a better idea. [frown]

Final thoughts...
At this point I'm not sure it is the same issue.
Can it be recreated with the same database on different computers? If so, then yes it's most likely data.
Then the question is, is it database size or corruption? This is harder to narrow down unless you have multiple large databases which have no cross contamination.
If you have this issue and have a relatively small database, it's most likely not performance.
Does it ever occur on different databases (for those of you who use more than one brain)?
Have you tried disabling all non-essential software (including antivirus)? Reinstalling Java?

It would also really help if everyone chimed in on if they used TheBrain 6x or not.

I'll end on a positive note. If it is performance Patrick mentioned in another post that the next version is supposed to improve the speed. So maybe that will resolve the issue.
macOS 10.14.6
TheBrain 10.0.54
Quote
Spacenexus
Zen

Thanks for the comprehensive notes on this. The same brain has been populated from version 1 of Personal Brain so the fact that it is still functioning after over 15 years with only mild senility says something i guess!

I only bring your case up since i tried using a brainzip about a year ago and recovering from that and the resulting brain was a mess. It turned out that the likely cause was using the same beta (probably around v5 or v6) that had caused your dataset to frag. Since my brainzip capability is non existant i never use them and my assumptions are that there is some relation to the freezes and the corruption that will have to remain in there somewhere, latent only until brainzip utility is raised (which will now be never in my case).

i notice delays on instant search in particular associated with constant disk activity. Just looking at the core file sizes:

LINKS.tbl: 2.4GB
THOUGHTS.tbl: 0.7GB
ATTACHMENTS.tbl: 0.5GB
ACTIVITYLOG.tbl: 0.3GB

These form part of the brain_cdbb directory that is almost 4GB in total.

If somehow Thebrain software has to query 4GB of files every time there is a change of thought activation (I don't have insight into the architecture so this is speculation - please correct me devs), then my assumption was that going to SSD would vastly improve performance. At current prices this would be a significant assumption to make so Bill's experience wasn't encouraging. We know that massive databases have had to address these issues and it would be great if v9 brings in some of these design cues into Thebrain architecture.

Thanks again for the helpful insights into your earlier issue.

Best

Jim







@181218:
TB8022 32bit
Java 32bit Version 8 Update 141

Firefox, Office 2013 Pro Plus 32bit
64bit Win10Pro
64bit Primary Laptop, 8GB RAM, Intel Core i7
64bit Secondary Laptop, 64GB RAM, Intel Xeon E3
Brain user since zygote
Quote
mctrexler
Thanks to both of you for these thoughts.  I will note that I see the mini freezes in multiple brains - at the very least my two large brains.  One of them has been going for many years, the other was started five years ago (I don't remember when TB6 was in play but probably after that).

I have tried running resmon.exe and watching to see if I can find a correlation between freezes and the various measures of CPU, disk, and memory activity that resmon demonstrates, but there's so much going on at the same time that it's challenging at best. 

I guess I had been assuming Jim's point as well - if TB has to search GB-size files for every new thought you'd think there would be a performance issue there.  That said, I have more than doubled the memory allocated to TB without any appreciable impact. 

Mark

TB8.0.1.1., Windows 7, Java 7_51
Quote
zenrain
Quote:
LINKS.tbl: 2.4GB
THOUGHTS.tbl: 0.7GB
ATTACHMENTS.tbl: 0.5GB
ACTIVITYLOG.tbl: 0.3GB


Uh, wow. That's a big database. [smile]
One other thing to add, I never had an problem creating a brainzip with the corrupt brain. It would stop when copying notes (and not due to a memory issue, it would stop on the same thought every time until I figured out which one was causing it).
I would guess the freezing issue at this point is more related to the size of the database and files than a database issue. Particularly if it's occurring when creating thoughts or doing thought modification rather than browsing. Most likely because it has to delve into that 2.4GB table and probably gets a bit exhausted just thinking about it... [wink]

A SSD may help slightly as access speed is faster, but it may not be the access speed that's slowing it down, so it may not (which with what Mark is saying is more likely). Then again, after moving to an SSD on my Mac, it pays off in so many other area's it's still worth it even if it doesn't help TheBrain performance.
macOS 10.14.6
TheBrain 10.0.54
Quote
Spacenexus
Is the inference the unthinkable...? That there is a fundamental architectural constraint to TheBrain that limits the size of the database. However, Jerry's brain's is over twice the size of mine but he doesn't hold attachments in them. Perhaps the search indexing is still an issue.

Matt, do you or the dev team have any information on performance of brains with 100k+ thoughts? Having said that Mark's smaller brains are also experiencing freezes although I do understand any number of issues could cause these problems.

I'll not forget the performance of Personal Brain v3 however. As a natively coded architecture i never experienced any delays or freezes, even though that was 10 years ago. I do remember noticing the drop in performance, freezes and latency with the switch to the Java based v4. That improved for a brief period at the beginning of v5 or v6 (I don't recall which) when there was a push for performance improvement, but since then it has more or less been every other thought there is a delay of some sort. I dare not it in presentations anymore as the intermittent latency can freeze the plex for tens of seconds sometimes.

I hope v9 blasts through the freeze.

Jim

@181218:
TB8022 32bit
Java 32bit Version 8 Update 141

Firefox, Office 2013 Pro Plus 32bit
64bit Win10Pro
64bit Primary Laptop, 8GB RAM, Intel Core i7
64bit Secondary Laptop, 64GB RAM, Intel Xeon E3
Brain user since zygote
Quote
mctrexler
Thus far I have not been afraid of using TB in presentations, but I do set up the presentation in advance so I don't have to use the search function (which is where I'm most likely to see the freeze when I'm not creating or modifying thoughts). 

In terms of Brain sizes as a potential indicator of freeze risk, my two Brains are 35,000 thoughts with 10,000 attachments and 15,000 thoughts with 10,000 attachments. The various table sizes per are between .2 and .4 of Jim's sizes referenced above. 

One freeze anecdote from 60 seconds ago. Created a new thought (instantaneous) typed in a word (instantaneous pop-up of that word in other thoughts), then cancelled the thought with the red X, and that led to a 10 second freeze (green link on the screen sitting there going nowhere since the thought was gone). Which is counter-intuitive in terms of when TB has to search files. 

Mark

TB8.0.1.1., Windows 7, Java 7_51
Quote
mctrexler
A quick clarification question from earlier in the thread.  Have other people implemented Matt's suggestion of assigning more memory to TB via

-Xmx320m[Return]

If so, do you see a change in TB's memory usage (Used and Free).  I would have thought that if you're assigning more memory to TB the sum of those two numbers would change from the default (about 190 MB) to whatever you've set.  But there is no change in the numbers, which makes me wonder whether the increased memory allocation has "taken." 

Any insights?

Mark

TB8.0.1.1., Windows 7, Java 7_51
Quote
Spacenexus
Thanks for the reminder Mark. I went insane and upgraded the memory allocation in vmoptions to:

-Xmx4096m

Will report back any performance hike.

Jim



@181218:
TB8022 32bit
Java 32bit Version 8 Update 141

Firefox, Office 2013 Pro Plus 32bit
64bit Win10Pro
64bit Primary Laptop, 8GB RAM, Intel Core i7
64bit Secondary Laptop, 64GB RAM, Intel Xeon E3
Brain user since zygote
Quote
Spacenexus
I decided to check my Java and recalled that it is 32bit. Is there a particular need to use 64bit Java with the 64bit TB installation? I'm not sure why I haven't previously considered this. Any thoughts / advice? I would have to upgrade my browsers to 64bit but that's not an issue really.

Jim
@181218:
TB8022 32bit
Java 32bit Version 8 Update 141

Firefox, Office 2013 Pro Plus 32bit
64bit Win10Pro
64bit Primary Laptop, 8GB RAM, Intel Core i7
64bit Secondary Laptop, 64GB RAM, Intel Xeon E3
Brain user since zygote
Quote

Newsletter Signup  Newsletter        Visit TheBrain Blog   Blog       Follow us on Twitter   Twitter       Like Us on Facebook   Facebook         Circle Us on Google+  Google         Watch Us on Youtube  YouTube       

TheBrain Mind Map & Mindmapping Software     Download TheBrain Mind Mapping Software