Bill Yang's Weblog

The simple and yet complex case of KISS

2012-01-09T14:23:00.000-08:00

I was blown away when first introduced to KISS principle. At the time, I was frustrated with how overly complicated our enterprise software was designed. It was a rewrite of a legacy system and our architect single-handedly wrote a custom framework on top of ASP.NET MVC framework, with the goal to make it more succinct and extensible.

The only problem, was that everybody was so confused about how it worked, that we lined up to the door of the architect with questions on how to implement particular logic. I carried that bitter taste of over-engineering for a long time, and have rigorously asked myself every time I make any design decisions, if my design was not simple enough...until now.

Today, a very smart and capable colleague criticized a simple piece of my code for being over-engineered, it was a surprise.The problem was fairly simple, so I skipped over things like IoC container, unit testing and the like, implementing only a very simple MVC design, with model layer logic located in separate project, using a single-class Micro ORM (PetaPoco) as opposed to a full-blown ORM.

In my mind, things cannot get simpler than this. Well, technically it can, like skipping the overhead of unit-testability, keeping all logic within single project, using built-in SqlDataReader instead of Micro ORM. However those measures only reduces complexity marginally, while making future refactoring exponentially more difficult. In my book, those are bad trade offs.

What I failed to recognize was that, you see, I have been writing code this way for many projects by now and all those concepts and code structures are intuitive to me. Writing and reading code structured this way is as straight-forward to me as code structured in the simplest possible way (TM).

This was not the case for my colleague.

He has different background than I do, and has grown different habits. All the "intuitive" project structure, the MicroORM, and unit testable code was extra complexity to him. While he recognizes the value of all those things, those are extra complexity that has no potential return due to simplicity of the project.

In the end, after some discussion with him and mental struggle with myself, I got rid of most of the "fluffy" stuff. I also lost the bitter taste for the "over-engineering" architect, after all, that whole framework is probably quite intuitive to him.

Lesson learned: Simplicity, like anything, is subjective -- what is simple to one person may not be the same for another. And when working in a team, you have to consider the whole team when making design decisions.

Visual Studio Website Project vs Web Application Project

2011-07-29T16:53:00.000-07:00

There probably has been billions of articles written about this topic, and I am no expert who understands the full extent on how the two types of project differ from each other. However, most of those articles are fairly old, and the year 2011 deserves a new article about this topic. :)

So if you are a developer working in a Microsoft shop, chances are you would use at least one of the following: ASP.NET MVC, NuGet, Unity Application Block, Unit Test, Gated Check-in, custom build event/script.

If you use any of those consistently in your projects, it's probably best to stick with Web Application Project, some of them (i.e. ASP.NET MVC) flat out cannot work with Website Project; some would, but requires non-typical setup which you have to investigate and assess.

Conclusion: in year 2011, if you are not sure, use Web Application Project.

CoffeeScript global variable and @ keyword

2011-07-28T22:59:00.000-07:00

I finally managed to pick up CoffeeScript, by writing a small game using CoffeeScript and Canvas. Writing code with CoffeeScript is a very fluent experience, even though I have never used Ruby and have C# background, the syntax feels so natural and code enjoyable to write. All the good things aside, I also got caught by a little gotcha of the language.

In the world of CoffeeScript, global variables must be explicitly declared, using either @ or window keyword, and most tutorials recommend @ due to its succinctness:

@luckyNumber = 4

this attaches luckyNumber into window object and thus accessible globally. The tricky thing is if you do the same declaration in a class:

class MyClass
  constructor: (@luckyNumber = 5) ->

all of sudden, @luckyNumber becomes a property of MyClass, and is not attached to the window object anymore! Now consider following situation:

@luckyNumber = 4
class MyClass 
  constructor: (@luckyNumber = 5) ->

  showLuckyNumber: () ->
    alert(@luckyNumber)

(new MyClass).showLuckyNumber()

Is it going to give you 4 or 5? ... Yes you probably guessed right, it's 5. To access the global variable, you need:

showLuckyNumber: () ->
    alert(window.luckyNumber)

While tricky, this is not exactly a CoffeeScript problem, but weirdness inherited from Javascript. Remember the golden rule: "It's just JavaScript".

A redemption of Waterfall model

2010-09-15T20:32:00.000-07:00

It is almost common knowledge now that waterfall model does not work, especially if you talk to programmers leaning to the agile persuasion. In this article I am going to give waterfall model its deserved credit, and to look at software development from a different perspective.

Before start, I need to first make a disclaimer -- I am not talking about "strict" waterfall or "strict" agile, I don't think those extreme cases apply to most real life situations.To me, any development processes that falls into defining clear phases (initiation, analysis, design, development, testing) are using waterfall model, regardless compromises made to accommodate different circumstances.

First, let's talk a little about history. Waterfall model originated from manufacturing and construction industries, the waterfall model does not only suits the limitation of the projects of those industries (where flexibility is impossible when real goods are involved), but also suits their management method -- The Command and Control Management Method.

If a company using command and control management method tries to use agile development model, guess what will happen? Well, agile practices rely heavily on making constant changes based on new discoveries and shifting requirements which will need to travel back to the top and come back down to the developers. When too many changes are pending, the decision maker becomes a bottleneck while the developers are held up until a decision can be made. This does not only creates more pressure on the decision maker but also would actually delay the project delivery.

On the other hand, waterfall model tends to minimize the communication requirement between decision maker and developers. When the project is in analysis/design phase, developers can be given other tasks like maintaining existing projects. Once the construction starts, developers are called in and start doing the work. There will be only occasional cases when changes are needed AND possible, giving the decision maker plenty leisure to examine issues popping up without delaying the project much.

Of course, the better solution would be to change the management process to be more decentralized to avoid decision making bottleneck. But unfortunately, real life situations are normally far from ideal. Maybe there aren't enough competent developers to delegate responsibilities to (and no, you don't want to set everybody lose to wreck havoc to the code base); or the company has been burnt by having too many fingers making decisions; or the business model can tolerate low quality software thus improving the development process is lower priority; or maybe simply just because.

You have to really think hard about the situation before applying any best practices or the hottest methodologies. Sometimes you may have to make compromises and take the second-best option.

Premature Abstraction

2010-08-09T20:43:00.000-07:00

Everyone knows that premature optimization is root of all evil and in extreme cases the programmer who does such thing will be stoned to death.

But not many people seem to be bothered with premature abstraction, or encapsulation or whatever people fancy it. Here is what I mean by premature abstraction: creating a train of complex framework to perform simple tasks to make them future proof.

This is a natural tendency after becoming more and more fluent in OO design, as one of the main advantage of OO is abstraction, to make code reusable and extensible. But like making a dish, if you put in too much of any sauce, it will ruin the taste. Abstraction is no exception.

Here is what will actually happen when you design something to be "future-proof": most of the anticipated requirements will be forgotten, and many new ones will show up from no where and on things you never thought of. When such thing happens, it'll be a painful experience to refactor that neatly designed and implemented framework to accommodate new requirements. If the project happen to be behind schedule at the time, which is always, guess what, you may not get the leisure and time to refactor the framework and hacks/workarounds will be applied to make it even harder to refactor. In fact once enough duct-tape is being applied to that originally neatly designed framework, nobody will dare to touch it any more.

Now I think about it, there is actually a proper name for it: it is called over-engineering. And it is quickly surpassing premature optimization to become the new root of all evil.

I will stone you next time I see you do that.

You don't need a publisher

2010-03-05T20:45:00.000-08:00

I was talking to my imaginative friend Joe the other day, trying to figure out why there still hasn't been a good website to read novels. We were talking about why book publishers should release books online in this era of Internet, and it struck me, why was publishers needed in the first place?

Back in the days, book publishing was a huge task and no small entities (a.k.a. authors) can do it for themselves, and in order to reach a wider audience without devoting their life to marketing which many may not be good at, book publishers were necessary.In fact things haven't changed much since! Book publishing is still a tremendous effort, you have to get editors to review the book, get advertisers to market the book, get designers to do book design, get printing companies to print the book, then go through a chain of distribution to get the books to readers.

And the result of that? Authors don't make that much. According to this post from a New York Times best selling author, she made a staggering $24,517.36 on a book that sold for 61,663 copies, about 40 cents for every book that's sold for $7.99 at retail. That's very little, but considering the amount of work the publisher has to do, it is only fair. Or is it?

In a business partnership, each partner has to contribute value to the business and take his/her share of profit based on the contribution, it is not always easily measurable, but mostly fair or the partnership won't work. Let's take a look what publisher brings to the table, that warrant a 95% cut [1].

Editor Review
Editors are quite helpful as they check for spelling, grammar and logical errors,they also give some good advices to improve the book or edit it directly.

Marketing
While authors can do a lot of the marketing themselves (and are increasingly required to), paid marketing campaign and advertisement still have their values. After all, who don't want to sell a few more books?

Printing and Distribution
These are normally done by separate companies. Printing and distribution is a lot of work, and book shelves are costly.

Publishers are indeed bringing a lot to the table in traditional publishing practice to warrant their share of profit. But things has changed! Advance in computer technology and the Internet has made a lot of things obsolete, including publishers, and here is why:

Editor Review
Spell checkers will weed out most of the typos and simple errors. Some helpful readers or a fellow author could definitely be able to help to do the proofreading to find any logical errors. In fact the readers will feel delighted to have a sneak peek into your newest book, it's a privilege! And even if errors manage to loom into your book, you can always go back and fix it, a luxury you don't have with paper backs.

And what beats reader feedback on how to improve your book? After all, the definition of a good book is that people like it enough that they will buy it.

Marketing
Authors already do a lot of marketing these days, and with the help of Internet, self-promotion has not been easier! If you really think you need some professional marketing, no one can stop you from hiring a marketing company, and you get to decide whether that's worth it.

Printing and Distribution
Need to say no more, distributing books online cost a fraction of traditional printing and distribution model. And there is always print-on-demand option for readers who prefer paperback.

Notes:
[1] To publishers' defense, they do not take 95% of the sales, money goes to all parties involved in the process of publishing a book. But to an author, it doesn't make a difference where the money went.

Google AI Challenge

2010-02-28T12:34:00.000-08:00

I have recently competed in Google AI Challenge, it was a lot of fun and I have learned quite a few things. Here is a quick explanation of the strategy I employ.

First it checks to see if two bots are seperated.
If two bots are connected, it will use Minimax/Alpha Beta Pruning to determine the best move it should take. The evaluation functions is as follows:
1, if I win: +1000; opponent wins: -1000
2, if disconnected, and I have more open spots: +800; opponent has more open spots: -800; same: 0; Open spots is determined by flood fill.
3, if still connected,
    find articulation points and divide map into zones
    find the largest zones
    use breadth first algorithm to divide the zones into spots I can reach first and spots enemy can reach first
    return final score as spots I can reach first - spots enemy can reach first
The whole Alpha Beta Pruning process is wrapped using a Iterative Deepening loop.

When two bots are seperated, simply use a breadth first algorithm to determine how many open spots are left after each move using flood fill. Then choose the move that leaves the most spots open, in case two moves are equally good, hug the wall.

It is written in C#, and can be found here: http://github.com/analyst74/Tron-Bot

Update: here is a great write up about the challenge and many related links http://www.benzedrine.cx/tron/

on Google vs China

2010-01-21T22:19:00.000-08:00

Now the immediate heat of discussion on Google's new approach to China has settled a little (until new news come out from either party).

A lot of interesting analysis and speculations have emerged from the blogsphere and news medias, from bashing China's (if you don't know yet) human rights problem to questioning Google's real motives. You can find them all here (courtesy to Google!).

One thing everyone agrees, is that this is a significant event, but how significant? Will it change the world?

Fast forward forty years.

It's year 2050, we have just achieved 80% CO2 reduction goal set out forty years ago thanks to innovations in energy sector. On the news:

(Yahoo News) A syndicate of Internet companies lead by prominent companies like Google, Microsoft and eBay, have threatened to blacklist Australia if a new tax is implemented. The tax will be applied to companies who do not voluntarily comply with its censoring rules, and cover costs associated with censoring on the ISP level. Rumors say proponents of the tax are rapidly losing support from voters for fears that those threats become real.

(Apple Daily) In a bold move to tackle lumping economy and increasing domestic unrest, Japanese government has announced a plan to lift regulations related to the Internet, citing "The Internet has sovereignty of its own and cannot be governed in traditional means". The plan includes establishing a governing entity called UN (United Netizens) and handing over all regulatory duties to this entity. UN council body will consist representatives from prominent Internet companies and user groups. Japan urges other countries to follow the "great leap forward in net-neutrality". Economic analysts say such move could attracts over 5 trillion foreign investment to flow to Japan, especially from countries with complex or repressive Internet laws like US and China.

How does this future sound?

Maybe, just maybe, one day the Internet could be free.

Two management styles

2009-12-06T16:10:00.000-08:00

There are two brilliant managers called Jack and Gordon.

They both work for the same company -- it's one of the nation's biggest corporation and has huge influence in the market. Having a job in the company is considered a privilege, let alone a managerial position. One has to be cream of the most brilliant crop to get there.

Jack is a very smart man, and he has gained substantial experience in this field to make him a distinguished expert in the company. This is not to say he lacks social skills though. In fact people admit that he posses both technical expertise and some political talent. He is a star leading a team of experts solving some of the company's most difficult problems. His talent is both widely appreciated by the senior management and admired by fellow employees, even those who are not in his team.

Gordon is also a very smart man, although not quite as sharp as Jack is. He is considered by most people to be a people's manager as compared to Jack, who is both a remarkable manager AND technical leader. Gordon's team is also one of the most successful teams in the company, the product they created has received outstanding reception and is bringing the company a steady stream of income measured in the millions. The success of Gordon, as people say, is widely contributed to the great idea (which did not come up by himself) and a superb team he got stuck with. His team is indeed superb, many of them have proved themselves to be A players and there is one young guy who is considered to be a wizard even among his team of A players, a star employee he is. Anyone who is lucky enough to stuck with a good idea and a superb team like Gordon's would have succeeded, probably more wildly than Gordon have.

Fast forward fifteen years.

Jack has gotten old and has stepped away from the company's core team since he lost his sharpness a while ago. The company still treats him well though, and he is enjoying his life with a much less stressful job. He misses the glory days, but nobody can fight time, his time has passed.

Gordon has lost the best members of his original team, three of them including the star employee were promoted "past" him, others eventually left and got promotion in other companies. But the team somehow managed to survive, he always seem to be lucky to recruit a few remarkable employees who kept the team and Gordon himself afloat, until about five years ago when he was promoted to senior management. "It is all due to his political talent", some people say.

But the company's chief architect disagrees:"He (Gordon) is the best manager I've ever met and the time working for him was the best of my career. He did not only understand both the business and technical sides, shielded us from senior management pressure, but also cared for us personally. He fought tooth and nails with HR to give us bigger raises and single handedly convinced senior management to promote me to the architect team. Without him, I wouldn't have been where I am now. I am so glad that I am reporting to him again."

The chief architect was not the only one Gordon promoted -- almost a dozen of the company's current managers used to work in Gordon's team and have since been recognized and promoted. They were equally thankful for the chance to work for Gordon. On the other hand, nobody seems to remember Jack except a few old timers, and Jack has never been "lucky" enough to recruit such talented people like Gordon has. Those "B players" who worked for him have either stayed where they were or have since left the company.

"Luck" is really important, as people say.

Does Inheritance breaks Encapsulation?

2009-11-09T20:24:00.000-08:00

Because inheritance exposes a subclass to details of its parent's implementation, it's often said that 'inheritance breaks encapsulation' (Gang of Four 1995:19)

No I did not read the GoF book, I'm quoting them because I believe quoting important people makes my point more correct.

So we know that inheritance breaks encapsulation because it exposes a subclass to details of its parent's implementation. But what is the harm of exposing parent's implementation to subclasses? Why does it matter?

Imagine this situation: you are coding a very core class for a system, and decided to make a field protected so it's convenient for subclasses to access it. After all, it is a core member field that all subclasses need to use and declaring it as protected seemed like reasonable thing to do. Everything worked fine and smoothly in the beginning. Then 5 years passed, assuming the system is actually making money for your company, it has grown substantially through a steady stream of feature requests. And many subclasses were created inheriting the core class you wrote. Now imagine you have to change that protected field, and *bam*, you're stuck! Because there are many subclasses sprang across many assemblies, and nobody has a headcount of them all! So you either end up not changing it, or do the change and fix as many subclasses as possible, then prey you don't miss any.

Does this sound familiar? Yes, this is the exact scenario when someone accidentally declared a field public for convenience when it should have been properly encapsulated, I hope it's not you, oops! (I'll be honest, I did this before!)

I believe this is the context when those influential people declared that inheritance breaks encapsulation. But a class does not have to expose its implementation to subclasses, take Hashtable for example: it is carefully designed to hide all implementation details from not just the rest of the world, but also its subclasses. So if there is a change to the default hashing method, how data is stored internally or any of the implementation details, all the subclasses created by developers across the world will not be broken! I believe a more proper statement would be:

Overusing protected members breaks encapsulation.

And the remedy to that? Don't overuse the protected keyword. :)

The problem of too many layers of indirection (abstraction)

2009-10-12T14:33:00.000-07:00

So everyone has probably heard this by now

All problems in computer science can be solved by another level of indirection

and the corollary

...except for the problem of too many layers of indirection

While it sounds smart and witty, I have never quite figured out what the corollary is referring to. After all, our society is built based on layers and layers of ~~indirection~~ abstraction, and that's how we advanced into modern society. It is a proven concept.

Well, let me step back a little and talk about an interesting issue I had recently. It was decided that our enterprise contract management software is not responsive enough and a few engineers were tasked to take a look at the problem.

Thanks to the advancement of software development, we now have awesome tools like dotTrace, among others, to simplify the daunting task of inserting time stamping instructions into every single method in the application. Looking at the profiling result, an interesting method quickly grabbed my attention -- it is a heavy lifter which makes up 40% of the overall page load time. Upon closer investigation, I realized that it is our new navigation tree which loads a gigantic metadata file that contains all information there is to know (like data dependency, context relation, data validation rule, permission rule and etc). To be more specific:

it was reloading the (static) data every page load.
only a small amount of data (those related to current page) is actually required.
it was executed twice on every page load.

Let's forget about the first two, and focus on the third problem. How did this happen? I mean, this looks like a simple problem that even junior programmers would know to avoid.
Well, the truth is, the original developer implemented the navigation module nicely, with caching and stuff, so the heavy-lifting method will never be called twice. Then a few months later, someone had to fix a caching bug -- the cached navigation tree became corrupted for some unknown reason. He looked around, and found a little method that was nicely packed and seems harmless, which will solve his problem by rebuilding the corrupted data. It was a perfectly logical choice on his side, although little did he know that about 3 abstraction layers down the road, it reads a gigantic xml file and create a few hundred objects on the fly.

Maybe the original developer should have documented this code better, maybe the other developer should have been more cautious when using other people's code. But the real issue is, abstraction hides so much detail that gives you a false sense of confidence. It makes you believe you know everything, after all, the method name and comment will be sufficient to describe what it does right? (hint, no) If it does, it would have to explain what all its function calls do, and the functions called by those functions it calls, and the functions called by those functions called by those functions it calls...

Every abstraction layer does not only adds a little over head to the CPU, but also to the poor human who has to read that code. Be careful, those little overheads may come and bite you one day.