Quality Software Engineering

Reaching The Technological Limits Of An Extension

So I’ve started work on some of the more data-centric DH stuff. Some limitations I feared when I came up with DH are now coming to pass.

For starters, it turns out indexing the keywords of every page a person visits can amount to a lot of data. I’m being sarcastic. I knew this from the jump. I had originally planned to cap the size of the database. The code is written but I never “turned it on.” I kind of wanted to see if capping was going to be necessary in the first place. Turns out (surprise) it is.

OK so the easy answer is to let people export the large databases for archival. This is where I am hitting the wall. Turns out Chrome doesn’t enjoy letting any single process have more than a couple MBs of data at a time. Combine this with the fact that Javascript (rightly) has no direct access to the user’s file system and you end up with a situation where exporting any significant amount of data from an IndexedDB data store become almost impossible without crashing the calling process.

Of course there are some server side things I could do but I’m not sure how comfortable people, or companies, would be with their browsing history being carted away to some dark server for processing.

I guess I’m going to have to sleep on this one. As with any problem attempted using a Turing-Complete  language, there IS a solution – but that’s never a guarantee that you’re going to like it.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s