Quality Software Engineering


I’ve been everywhere

I’ve decided to go on this random adventure. Over the years I’ve explored a couple of different CS areas. First I started out with advanced JS programming. Eventually I realized it was pretty hard to break new ground there. I’m proud of the things I built and the knowledge I gained but I realized that if one want to break new ground, one has to go where the hard stuff is.

I moved on to machine learning. That’s was cool but I realized early on that breaking new ground there meant a deep understanding of mathematics. I bought some math books and bulked up on math. At a certain point I felt math wasn’t really what I was interested in – at least not the level of math necessary to come up with novel machine learning angles.

My ML adventures also turned out to be worth it because now I can say I have a solid basic understanding of ML in theory and practice, not to mention I feel pretty comfortable with math-centric academic papers as long as they don’t get to domain specific.

I suppose the number one thing I learned is that I like reading and writing software. I wondered where the hard stuff lie for a person with my interests. I settled on operating systems. OK now what?

It turns out OS development is a bit of a dark art. There’s resource out there, but they are all over the place and vary wildly in quality. I’ve always wanted to know how to make cool stuff from scratch. I’m now trying to learn how to bring dead metal and silicon to life via low-level programming. At some point this may lead me to hardware development. At the bootloader level, you need to be very aware of the hardware and there aren’t that many abstractions between you and it.

I started off by reading a kernel development manual from 1999. I know it’s outdated but I figured you have start with the basics. It’s worked out pretty good because it was a mash up of generic OS concepts with Linux-specific examples.

Next I moved on the the Kernel Hacker’s Guide (1998) which turned out not to be so great but it did point me in a good direction. It talked about how system calls worked in Linux. Although this is no longer how they work, the code and concepts it pointed to dated back to the earliest days which provided great insight into the thought process of Linus and other early contributors. I also started looking at source and finding parts that were referenced had been removed from the kernel, which also provided insight. Why had these parts been removed? What had they been replaced with?

Currently I’m reading a slightly more recent kernel guide, 2004’s Linux Kernel 2.4 Internals. This also referenced a since-excised piece of code: the original bootsect.S boot loader written by the man himself, Linus. This in turn pushed me to learn about the ancient ways of gas and also how pre-page table and global descriptor table (GDT) memory segmentation worked.






Long time no see

Wow wordpress.com has changed a lot since the last time I blogged. I’m on a very different journey. While I had a lot of fun with my JS projects, I came to a point where I didn’t feel there was much room to be truly innovative. I also saw a future where many of the projects (like react and angular) would be swallowed up by the ever-evolving DOM and JS API.

In my absence, I found out about hackernews and it’s really broadened my horizons. Mostly HN got me much more interested in the lower-levels of software engineering.

I also went about a 3rd of the way through Andrew Ng’s machine learning course. that was really cool and I enjoy having a basic understanding of how it works. I did however learn that although I tolerate math, it’s not really my passion. I didn’t see any way to innovate in ML without a very deep understand of calculus and proofs. So I moved on.

What I’ve learned about my self is that I like building software. I like creating and constructing the puzzle that is complicated software. If I need to understand algorithms to make that software efficient,  I’m cool with that. Do I want to invent new algos? No.

Going back to my days as a musician, I never liked using samples. I always liked to make my songs from scratch. I’ve found I have the same proclivity with software. But now I want to be able to bring dead metal to life. I want to be able to put a ghost inside the machine – on operating system.

So now I’m studying the Linux kernel. My goal is to be able to write a small Linux distro in rust. I plan to use this blog to document my exploration.

I think I will start with implementing ext2 in Rust.


The grand vision for structureJS

I have significantly fleshed out this site with content and I am quite proud of how far the project has come along. After trying my hand, ever so briefly, at being in a start up, I feel much more at home with a project like structureJS.  Something  by developers, for developers.

I have virtually no chance at getting rich with this project but I have a good chance at getting something more important – the respect of my peers.

Part I – On-Demand Complexity

This project was obviously inspired by RequireJS. What I set out to create was a tool that gives you the core functionality of RJS, but with the complexity ratcheted down. RJS is a robust and powerful tool. I am sure there are projects out there using every bit of its most advanced functionality, but I found myself being put off by the complexity of RJS when I really only needed it to do one task – load my script in the correct order.

I believe strongly in on-demand complexity. This is why I built the structureJS core in a modular fashion. My hope is that other devs who need some of the more advanced functionality of RJS will write modules that implement those features, share those modules with the community, and whoever needs those features can plug those module into their structureJS core. For everyone else, the core remains basic. The idea is that this allows keeps the barrier to entry as low as possible.

Part II – A Module Community

So the final component to my vision, is to remove the need to download anything to try out JS tools written by other people.  

This is why structreJS has the reserved directory alias “remote.” Using that alias, a dev can inject another dev’s modules into their project and give them a spin. If it works out, then download the module and add it to your project. If not, just remove that module from your manifest.

I would like to create a module sharing site that lets devs comment on a rate modules. I hope that eventually structureJS will enable the JS community at large to create a vast library of easily accessible tools for anyone to try out and use. Want to use client-side encryption in your app? There’s a module for that.

Want to use IndexedDB but scared of all the crazy async stuff. There’s a module for that too.

I hope that making it easy to use modules that provide advanced functionality to your application will result in faster, more powerful, and more efficient development of the next wave of web applications.

Anyways, that’s where my head is at with this project. I hope I can make this vision a reality. 


Organizing Deeper History Using structureJS

I am a manic creator. I get an idea. Usually in the shower, the bathroom, or in bed and at the next chance I start working furiously on it until I reach a state where I feel the idea has become a reality. Some ideas, like Deeper History, are actually worth continuing to work on.

But when the mania is gone it becomes hard to pick my instrument (keyboard) back up-at least until my next creative surge. Well I’m in one now and DH has benefited greatly. 

In the beginning, I work on pure creative adrenaline. “Quick & Dirty” is the best way to describe the syntactic results of such an endeavor. If the project is worthy, eventually I need to smooth out those rough edges. This is called refactoring, and boy and girls it is NO FUN. 

One of my big problems with DH is that it was broken up into a # of files using a project I made called MMCD. While I had good intentions with MMCD, it turned out to be bloated and, most importantly, stood in the way of minification –  something the growing DH is desperately in need of.

So in my infinite wisdom I decided to remake MMCD into something that didn’t suck and I think I succeeded. Taking all of the lessons of MMCD I managed to make something I am most proud of. It’s called structureJS. It does the following:

  1. Break your app up into seperate files.
  2. Create reusable, configurable modules.
  3. Declare the structure of the app in any order and structureJS will resolve order dependency automatically.
  4. Minify and combine all modules and files into a single file right from your browser.
  5. Utilize module dependencies easily and add semantic meaning to your modules
  6. Only puts one variable into the global namespace
  7. Partially AMD compliant (works with jQuery AMD for now, more to come)

I started a separate blog to talk about structureJS (http://structurejs.wordpress.com/). You can also get it from GitHub (https://github.com/ShamariFeaster/structureJS)

Anyways, now DH is properly organized and minified. I am still working out some things with the massive new update in the pipeline so it’s probably going to be another couple of weeks before releasing DH. Anyways, I just wanted to share structureJS and a couple of thougts with you. Cheers!


Much Inspiration, Wow

Inspiration to do art comes in spurts and I guess I’m in one now. In the last 72 hours I have:

  1. Implemented encryption on the DH database
  2. Written a small Javascript unit testing library a la QUnit
  3. Wrote a super sweet IndexedDb API
  4. Reduced noise in the DH database to < 1%
  5. Written an algorithm I actually think is worth patenting


I’m probably jinxing myself here, but I’ve finally broken through, and stayed about, the 2,500 user ceiling. I’ve been proud of DH up to this point but the next major release is going to be my best work ever.

These improvements will make DH into a serious tool for professional researchers.

DH Is Getting A New And Improved Compression Algorithm

I will be releasing DH with a new algorithm I wrote that improves the data compression of your history content by about 50%. I’m calling it an adaptive cache compression algorithm. 

Also the old compression algorithm has been improved to reduce or eliminate “noise” from sites like Facebook and Google Drive that inject a lot of data into their pages.

Lastly, I’ve decided to implement a site black list. I was not in favor of this before because I refuse to implement non-general solutions, but with Google’s search page my reasoning is two-fold. 

  1. Why would anyone recall the content of a search page? You would just Google it again – it’s faster.
  2. The value of indexed content from this page was so little, yet it’s database footprint was so large.

So after looking at those two reasons I decided, in this case, a “fitted” solution was warranted.


Reaching The Technological Limits Of An Extension

So I’ve started work on some of the more data-centric DH stuff. Some limitations I feared when I came up with DH are now coming to pass.

For starters, it turns out indexing the keywords of every page a person visits can amount to a lot of data. I’m being sarcastic. I knew this from the jump. I had originally planned to cap the size of the database. The code is written but I never “turned it on.” I kind of wanted to see if capping was going to be necessary in the first place. Turns out (surprise) it is.

OK so the easy answer is to let people export the large databases for archival. This is where I am hitting the wall. Turns out Chrome doesn’t enjoy letting any single process have more than a couple MBs of data at a time. Combine this with the fact that Javascript (rightly) has no direct access to the user’s file system and you end up with a situation where exporting any significant amount of data from an IndexedDB data store become almost impossible without crashing the calling process.

Of course there are some server side things I could do but I’m not sure how comfortable people, or companies, would be with their browsing history being carted away to some dark server for processing.

I guess I’m going to have to sleep on this one. As with any problem attempted using a Turing-Complete  language, there IS a solution – but that’s never a guarantee that you’re going to like it.