Lightning Fast Filtering in JavaScript

I thought I'd share a technique for filtering large sets of data quickly in a web page. The scenario is a biggish set of data, 100,000 records, with each record having 3 properties, and we want to filter them using text matching.

Here is the finished article with dummy generated data. Give it a try. Just type any 3 or 4 characters and watch it do its thing.

See the Pen Fast Filtering with JavaScript by Chris Smith (@chris22smith) on CodePen.

Fast, isn't it? You can go into the JavaScript tab and see how it all works but I'll explain some of the main parts.

When trying to make it fast the most important thing to be aware of is that changing JavaScript objects is fast, changing the DOM is slow. So, we do as much as we can in JavaScript and then just update the DOM at the end. If you try to create 100,000 <li> tags it will be very slow and will probably die on you.

The first step is to set up a function to convert the data array into a HTML list. We keep this fast by setting a limit on how many items we will show. In this demo I've set it to 10 but even increasing it to 100 makes little difference to the speed. They key thing is not to get into big numbers which put a strain on the DOM. So we don't loop through all the items, only the first few. A nice small loop, which runs quickly.

The building is all done in JavaScript, with nothing touching the DOM until we have a complete Document Fragment with all the list items ready to append to our list. I've separated out the building of each list item into its own function to hopefully make this even faster. Whenever iterating through large numbers of records the more variables and functions we can take out of the loop and pre-define the better.

The next step is to get the data array with the full data and filter it down into a smaller array using the native JavaScript filter() method. This method is incredibly fast, much quicker than using a for loop to iterate through each item. You can read more about filter() on MDN. If you use a predefined function with it rather than an anonymous function it's faster still.

Once we have the filtered down array we can call our function to rebuild the list in HTML. It all happens in the blink of an eye.

As a final touch I've added a count message function which gets called each time we filter. This just helps to see what's going on - how many records are being shown and the size of the reduced data set.