Data Limitations?

What can you give as some guidance as to what limits should be placed on the amount of data attempting to be displayed in the grid?

Is there some limit to how large the javascript array variable can be...

I may want to store thousands of rows of 200 bytes or so each. Is this an issue (codewise)?

In limiting testing I got it to work well (pretty and everything!) with smaller sets of data and under Mozilla Firebird. Could not get it to work yet (identical page and data) under I.E.

Under any circumstances this project rocks!

As far as I know javascript arrays can be VERY large and there is no hardcoded limitations in the code, so codewise this is not an issue.

However javascript performance, browser parsing and rendering performance - is a serious issue. The only possible solution here is to minimize the amount of processing at the client-side, for example, only request and render the data within the visible part of the scrolling area. The ActiveWidgets framework is designed exactly for that, but this early grid prototype is not using any of the advanced features and simply renders all available data. With this simple approach the realistic limit is 100-1000 rows (1000-10000 cells, 0.5-5 sec wait).

For higher volumes of data it is necessary to implement selective rendering and probably paged data retrieval. Then you can go up to 10000-100000 rows. It may also require different approach to sorting (re-request properly sorted data from the server/database). Some of that will be implemented before the final release but its difficult to give any more details now.

Just today I tried the grid with a large xml dataset - 150 cols x 20 rows. Firefox prompted me to cancel the script twice but eventually finished and everything was fine. Unfortunately IE 5.5 (my target browser) quit responding altogether and never finished rendering the page. =(

I'm not sure what I'm going to do about it yet since my application needs to handle very wide datasets (I'm already paging the rows). If anyone else has overcome this hurdle I'd love to know how.

Yes, I noticed that IE rendering performance is much worse when you have large number of columns compared to the large number of rows (with the same total number of cells). I don't know the solution for this. I would say at least make sure it is rendering and not XPath calculations or other XML processing (i.e. compare to plain JS array data).

Thanks for your response, Alex.

I switched to the JS array and while I notice an improvement in speed in general, IE 5.5 has trouble with a smaller dataset (63 columns). It renders the grid but then it sits there for a few minutes (I stopped watching after 2 full minutes) before it returns control. With Firefox and the larger dataset (150 columns) I don't even get prompted to cancel the script like I was before. So, the improvement is there and it's noticeable but it seems I'm worse off than before with IE 5.5 and wide datasets. Unfortunately, my target audience is IE 5.5 (corporate browser standard - ugh) so I am still stuck with this issue.

I'm curious, what is the widget doing after it displays the grid but before it returns control and displays the scrollbars? That seems to be the choke point, at least visually.

I can make my JS available if anyone wants to run tests.

I would say the widget probably doesn't do anything in your case - it is IE rendering which makes a problem (i.e. my stylesheet, your page layout or its combination). Can you run a test as described here http://www.activewidgets.com/messages/897-1.htm and check which phase is the longest one?

You are correct. After running the numbers as you suggested it is the rendering part that is taking over 2 minutes for my 63-column dataset. I'm running this on a P2 400MHz laptop. I tried it in IE6 on a P3 1GHz machine and the last part was still a distinct choke point at over 1 minute. Forget about trying to view the 150-column dataset at this point. =((( On my development machine (P4 2.5GHz) with Firefox the same dataset takes 147 ms to render - less than 2/10 of a second! I'm running linux so I can't see IE on the same machine.

Great product, though. I'm thoroughly impressed!

Another suggestion - try to run the test on an empty page (grid only) and also fix the size of the grid (say 800px by 400px). If this runs ok - then the problem might be between the grid and your page layout (if you are using tables - 'table-layout: fixed' may help or maybe you have to set the size of the grid explicitly).

Hi. Thanks for responding.

A page with just the grid (63 columns) yields better yet still unsatisfactory results - over 1.5 mins on the P2 and a little over 1 min on the P3. I'm not using any tables in my layout, the bulk of my page is the grid with a few navigation lists. Setting the grid size explicitly to 800x400 (using css) yields minimal, if any, improvement.

What I find odd is that the page will display very quickly with the viewable region but then there is a long delay before it finishes. As if it knows how much is visible and will display that portion while it finishes what's not immediately viewable. My users will not be satisfied to wait even 20 seconds for the grid to display, much less 2 minutes. They will have to live without the nice features of your grid for now.

I appreciate your efforts. Thanks! =)

I just wanted to follow this up with the results of a recent test I did...

On my P2 400MHz laptop using Firefox 1.0PR, the grid containing a very wide dataset renders in 8 seconds compared to 108 seconds using IE5.5 on the same computer. I guess that illustrates the deficiencies in IE's rendering engine? ;-)

We did some tests here (Firefox browser) with ActiveWidgets doing the sorting, and performance dropped off significantly after 2500 rows with columns=4... After that it was too slow. Just fyi.