• Posts 44
  • Member Since
  • Last Active
Email
All Posts Topics Started
PB 4.5.0.8 Updating Database Hangs
Yes, it could happen.

I tried mine on a second computer with 4GB physical memory, I also allocated 1GB to VM, however, a lower CPU (Duo Core vs Quad Core in my first computer), PB hang as well.

So I confirm with your findings.

I am awaiting a newer version to address this issue. (Memory problem is difficult to address, we may have to give them good amount of time to find the fix)
XML Import broke in 4.5.0.3?
Take a look at here:

http://forums.thebrain.com/tool/post/thebrain/vpost?id=2846131


Also, in the second computer that I have tried, not importing XML anymore, simply take the brainzip file which I have created in the first computer and open it in the second computer.

Edit: After the importing and indexing finish, my "brain" is using much less memory now. :-)


XML Import broke in 4.5.0.3?
I brainzipped this, and send this to a different computer, which also has 4GB ram, but less fortunately, only a Duo Core machine, guess what, the CPU consumption for PB went up to 100%. So obviously, simply more memory isn't helping either.

I am waiting for a newer version now, hopefully with one of the following fixes:

1. Better Memory Management and less CPU consumption in dealing with large brains. (Most important priority) - Otherwise many folks which have large brains won't be able to uprade to the new version.

2. Allow better incremental update indexing (at least the status showing consistently). This will allow (at least in my case) me to slice up the importing into much smaller chunks and in the end, using less memory and CPU (hopefully) for indexing.

PB 4.5.0.8 Updating Database Hangs
You may need to try a much bigger memory assignment. In my case, I knew that indexing failed (when I assigned 550MB Ram to PB VM) for a brain which has approximately 25,000 thoughts in it.

I increased memory assignment significantly to 1GB for a brain which has over 42,000 thoughts and 120,000 links for the indexing to finish. But I have a quad core cpu.

Change to following

-Xmx196m

to

-Xmx1000m

For 1GB assignment. You may also try

-Xmx775

It may work for you.
Visual effect suggestion
By default, PB VM only uses certain amount of memory no matter how much memory you might have on your system. On my machine, I have 4GB installed, I have to manually enable 1GB assigned to VM. See my other message on XML import for details on how to assign more memory to your VM.

Edit: BTW, it`s only an educated guess, with your number thoughts being not very large, I do suspect there is a memory management issue on PB`s part.

Visual effect suggestion
Could be a memory problem. I am experiencing this by importing XML files. Currently using 1GB Memory assigned to PB VM, when using 550MB Memory, the indexing failed. Just a note, this XML contains approximately 42000 thoughts and 120000 links.


XML Import broke in 4.5.0.3?
First try, it finished importing all 42,372 thoughts, and around 120,000 links, then it got stuck while updating the thought properties. Memory usage is around 500MB.

I am going to increase the memory to PB VM a bit more to see whether that will make a difference

Second try, PB VM=450m, it got stuck at the same place

Third try, PB VM=550m, reduce the number books to 66 (OT and NT), it got imported successfully. It`s doing the indexing in the background now. - Edit: Unfortunately, the indexing got stuck at book 18. I see PB now using 650M ram, that means memory probably exhausted.

I will do a fourth run, with PB VM=1000m and for all the books (OT, NT and Apocrypha). Edit: Importing successful, now doing the indexing, we shall see whether this will be enough memory. - update, indexing is progressing to book 71 (12 more to go), however, my system is showing PB is using 1GB, hope it`s not trying to use over that, otherwise, may crash again. :-(

Finally, by assigning 1GB memory, I was able to finish the indexing without crashing.

For Harlan: I still think there is a very poor management of memory usage in dealing with indexing. It`s probably not acceptable to have to use over 1GB in order to sort this brain, even though it has over 42,000 thoughts and 120,000 links.

Click image for larger version - Name: PB_uses_over_1GB.JPG, Views: 145, Size: 14.72 KB Click image for larger version - Name: PB_uses_over_1GB_1.JPG, Views: 144, Size: 14.82 KB
XML Import broke in 4.5.0.3?
In my individual book importing, my each book has two abstract children thoughts, as an example

Genesis

Genesis Chapters                   Genesis Pericopes

Then all the chapters will be children of Genesis Chapters thought, and all the pericopes will be children of Genesis Pericopes thought.

Then each verse will have exactly two parents, one the chapter thought this verse belongs in, one the pericope thought this verse belongs in.

There are jump thoughts (as we discussed several weeks ago) on each level for easy nagivation.

And I haven't noticed the same problem you have experienced, the book importing will have (again, as an example) genesis linked to the current active thought in the brain.

I could guess the following might be possible cause:

1. the order of the thoughts in your xml file?
2. the order of guids?

I am regenerating all the books at this stage, I added the framework for the overall bible structure - which I did not need initially as the book are imported individually. It will take 20 minutes to finish. Then I can import them all at once. It will be nice to have the structure built-in.

Right now, my script is on book 67 (the 1st book of Apocrypha)

Should be done soon.



XML Import broke in 4.5.0.3?
Oh, yes.

What can I do is to combine all my xml files into one file and import. I will let you know whether that will help in my case.

I guess the reason I have never combined them in the first place was prior to the new approach, it will be too much to copy all 200MB of xml source in one shot. Hope this time this problem will be solved.

Cheers

XML Import broke in 4.5.0.3?
mbaas,

Thanks for your message.

You wrote, "So now I tried to generate a smaller file (4bks) and that has been imported fine, but the indexing was horror - consumed CPU for >2hrs, until I had to kill the .exe (needed my CPU for other work)."

When you say the indexing was horror, could you please clarify that you are doing the indexing while importing individual files or you are doing the re-indexing after all the small files being imported?

Also, if you have not already tried, please give your PB virtual machine more memory by using the following tip (originally given to Br. K by Harlan):
-------------------------------------------------------------------------------

To increase the available memory (when operations such as importing a large PB3 file are failing)

1. Close PB4

2. Create a text file and put the following content into it:

-Xmx196m

Make sure there is a linefeed (Enter) at the of the text. Save this file and name it PersonalBrain.vmoptions in the PB program folder (the same location where the output.log file appears).

3. Restart PB4.

---------------------------------------------------------------------------------

Let me know whether that makes a difference for you or not. (In my case, I tried -Xmx384m, since I have lots of memory to spare)

In my case, while importing individual files, the indexing was not done consistently - because of the lack of a consistent visual clue as to when the index should be finished - so my gui-script does not know how long it should wait for the index to finish. This is one part of the problem. The second part, after all files being imported, I can do the re-indexing, however, as I mentioned, the re-indexing did nothing at all, all the missing indexings are not picked up by the re-indexing process.

BTW, our NRSV version has three layers of contents duplications:

We have Chapter Layer Thoughts which contain contents for each chapter, we have Pericope Layer Thoughts which contain contents for each pericope, we also have verse layer thoughts which contain verse content.

You also wrote, "One thing I observed during the import was that it seems the plex was continously redrawn - I have pasted the thought below the home-thought which has 2 jumps, and these jumps were always redrawn."

I haven't noticed this behaviour. One thing I did notice, I wrote a gui-script to radomly walk the bible (different from wander, as wander did not activate the thoughts it walked through), this script is very very simple, all it does is to use the arrow keys (up, down, left, right) and enter to walk through the thoughts. It actually on one occasion crashed my computer, on second occasion increased my memory consumption to around 500MB (after one hour of random walk). This is a big problem, but probably difficult to diagonose.

If you need any help with testing your importing, I can help you. As my gui-script is pretty good at doing this. What it does is it will ran through all my xml files one at a time, and copy and paste it into the PB, then wait for the status screen (copying thoughts, copying links, updating thoughts...) to disappear, then continue the next. - the missing link here is the status screen does not consistently show as a last step updating indexes.  So, the indexing part will have to wait until they have fixed the problems.







XML Import broke in 4.5.0.3?
Upon further trials, I was able to find more clues to the problem:

1. The visual status of updating the index only appears once in a blue moon, however, on one occasion, I was able to do a screen shot of it. The following screen shot provides further proof of it. But as I said, I have no idea what's triggering it. It would be great it could consistently showing this status.

2. If I only import one XML files, and wait (even if there is no status showing index is being done) indefinite amount of time, the index could eventually pick up. But in this way, how long I need to wait in order to import all 83 files?

3. Upon consecutive importing of 83 files by inspecting the status (without the visual clue of updating index, as a result, not waiting for indexing finish), those index was never able to complete, even if I do the re-indexing at a later stage, still not successful in indexing.

So the best solution is to consistently  provide the visual clue of updating indexing, so my script could wait the right amount of time to import the XML files.

For now, I am going to change my script to wait a long time after finishing importing one file to see whether there is any progress in terms of indexing.

A large brain without the indexing is a nightmare to use.
This brain might as well does not exist.

Click image for larger version - Name: brain_pasting_thoughts.JPG, Views: 240, Size: 96.85 KB
XML Import broke in 4.5.0.3?
Harlan,

I think the mechanism is there, I guess definitely it's bug. I wrote a Gui-script to automate the importing process (by watching the appearing and disappearing of the status screen.) since I have 83 files to import.

There are multiple things I have experienced:

1. copying is not doing the indexing. (I knew in one of occasions, it was showing in the status screen that it is doing the indexing) but on all the later tries, it was not. And I knew the indexing was not done since those newly copied thoughts are not being picked up when doing the search.

2. re-indexing is not helping to address the issue, by applying your suggestions to give PB more memory, I applied 384m to the PB virtual machine, on my quad core cpu machine, it was able to index the brain (total thoughts ~42,000), however, when doing the search, many of the thoughts did not show up, upon further inspection, only activated thoughts were picked up by search. So I guess re-index also failed, even if you applied more memory to PB VM, also have a faster CPU. So this is another bug.

3. Huge Memory consumption and hang. I left the PB on overnight, and I created another GUI-script to randomly walk (not through the wander, as it will not activate the thought) . In the morning, when I checked, the machine crashed. All the script was doing is using the arrow keys(up, down, left, right) and enter to activate the thoughts, I was hoping after running overnight, it will pick up most of the thoughts, but instead, it crashed my machine. When I re-ran the script this morning, after 1 hour, I noticed that PB is using ~500MB of ram. So I stopped the script.  After these two runs, I had 7130 (out of 42000) thoughts are activated now.

4. Since the work I am doing involved copyrighted materials, I cannot upload them here for public viewing). However, I am willing to share with your development team privately if that's helpful for your team. I can send you 83 XML packed in one rar files (around 5MB), it will uncompressed to around 200MB. After importing into the brain, the brainzipped version will be around 33MB, so definitely the XML version will be ideal. Also, it will help you stress test the robustness of XML importing and iron out any bug you might have.

If you are interested, I can upload the rar archive here by creating a password, and I can PM you the password. Let me know whether this is suitable for you.
XML Import broke in 4.5.0.3?
Harlan,

Thanks for the new version PB4.5.0.8. I have been using the following to import XML files now:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE BrainData SYSTEM "http://www.thebrain.com/dtd/BrainData1.dtd">
<BrainData>
    <BrainSourceLocation>xml_file_name_here</BrainSourceLocation>
</BrainData>
On one of the occassions, I noticed that copying process is doing the indexing of the imported file as well. I got excited, thought this is a new feature, because as you can imagine, if you wait after all the files (83 of them) are imported, then do indexing, it's a nightmare, first of all, it takes forever, secondly, the index process does not work. However, I could not make the indexing happen again on later tries, I am quite sure it was there once. Could you please let me know if there is a trigger to make the copying do the indexing of the imported thoughts from XML? If not, could we add one in, it's probably better to do incremental indexing rather than doing it as a whole in the end.

Secondly, could we expand the above structure to import more than one at the same time, such as the following:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE BrainData SYSTEM "http://www.thebrain.com/dtd/BrainData1.dtd">
<BrainData>
    <BrainSourceLocation>xml_file1_name_here</BrainSourceLocation>
   <BrainSourceLocation>xml_file2_name_here</BrainSourceLocation>
   <BrainSourceLocation>xml_file3_name_here</BrainSourceLocation>
...
</BrainData>

Currently, if I did the above, it will only do the first file, and ignore the rest.


4.5.0.7 - Release Candidate
Just noticed the high memory and cpu usage of the new release.
PB 2008 wish list / priority list
punkmonksf,

You wrote: "Next is to figure out how to eliminate the need for re-duplication of data. Because of PB's lack of interaction with other programs, this virtually ensures that data will have to be created and maintained in more than one place."

You can already export via copy thoughts function of your brain as an XML file. There are many tools available to process XML files, hence this will provide a bridge to other programs that also "speaks" XML. On the other hand, one can also construct XML files (as an example, I have created bible structures by writing a simple program to auto-generate the XML files) automatically. This is useful for not so "organic" information, ie, information which are well studied and structured, and you would like them to be included in your brain. For example, bible, World History, Geography, Classic Music, etc.

I am also in the midst of contructing a brain thoughtlet, which is a GUI written in autoahk, which in its current stage, can auto extract some XML information from your opened brains through some hotkeys, reveal and copy your thought guid/thought name, analyze the link structure. It can also auto open other linked brains (ie, brains not currently open but you wish to), you can do this all in this little brainlet (I use the name brain spy, in rhythm with window spy, but I am not so sure it's a good name), in the future, I would like it to be able to auto combine two or more brains through some predefined anchor thought. This thoughtlet has been in development for last two weeks.

Anyway, just my two cents.

count post selected

Add a Website Forum to your website.

Newsletter Signup  Newsletter Signup        Visit TheBrain Blog   Visit TheBrain Blog       Follow us on Twitter   Follow Us       Like Us on Facebook   Like Us         Circle Us on Google+  Circle Us         Watch Us on Youtube  Watch Us       

TheBrain Mind Map & Mindmapping Software     Download TheBrain Mind Mapping Software