What I’ve been finding is that the quality of documentation for developers isn’t good, and is both inconsistent and frequently missing important details (or hasn’t been updated to reflect changes).

I’m going to post a few things I’ve learned over the past few days that took far more time to figure out than it should have. Hopefully somebody else will benefit from my wasted time.

1. Closing a toolbar.

The toolbar that I’m working on is quite thick, and takes up a lot of real estate. In order to make it more user friendly, my customer wanted a close button to be incorporated. The custom toolbars that I’ve used in the past didn’t have such a feature, and I couldn’t just look and see what they do.

The key was that the close functionality needed to perform the same command that clicking on the toolbar’s menu item under View -> Toolbars would do – otherwise if it just hides the bar, it would be hard for a user to reopen it later on.

The code that I eventually found is as follows:

var toolbar = document.getElementById(“YOURNAME-Toolbar”);
var visibility = toolbar.collapsed;
setToolbarVisibility(toolbar, visibility);

Note that “YOURNAME-Toolbar” is the unique id of the <toolbar> element in your XUL overlay (that’s not actually obvious, because that element “lives” inside of a <toolbox> element and an <overlay> element. Closing, collapsing or hiding either of those really messes up the toolbar.

I’ve been unable to find documentation on developer.mozilla.org for the setToolbarVisibility function; finding this was painful.

2. Changing the arrow images in an <arrowscrollbox> element

The <arrowscrollbox> element is very useful in a number of circumstances. Unfortunately the arrow icons are sufficiently small that users may not notice them. They’re also too close to the contents of the box by default.

I couldn’t find a single source of documentation for the css styles that target this element. The following took hours of hunting around – and experimentation:

#YOURARROWBOX .autorepeatbutton-up[orient="horizontal"],
#YOURARROWBOX .autorepeatbutton-down[orient="horizontal"][chromedir="rtl"],
#YOURARROWBOX .scrollbutton-up[orient="horizontal"],
#YOURARROWBOX .scrollbutton-down[orient="horizontal"][chromedir="rtl"] {
list-style-image: url(“chrome://ud/skin/scroll_left.png”);
}

#YOURARROWBOX .autorepeatbutton-down[orient="horizontal"],
#YOURARROWBOX .autorepeatbutton-up[orient="horizontal"][chromedir="rtl"],
#YOURARROWBOX .scrollbutton-down[orient="horizontal"],
#YOURARROWBOX .scrollbutton-up[orient="horizontal"][chromedir="rtl"] {
list-style-image: url(“chrome://ud/skin/scroll_right.png”);
}

Note that #YOURARROWBOX is the unique id of the container – if you don’t put an id on it, you could do odd things to the rest of your Firefox theme. It looks like the arrow buttons are implemented something like a bulleted list, in the sense that they use list-style-image to set the image that displays. The urls in this case are internal to your plugin.

Also note that “up” and “down” in this case (horizontal scrollbar) mean left and right. It looks like the same css selectors are used.

3. When in doubt, look at other people’s code

Toolbars are kept in your Firefox profile folder, which is usually (on Windows anyhow) somewhere under [your home]/AppData/Roaming/Mozilla/Firefox/Profiles/[some folder]/extensions/.

The extensions themselves are .XPI files, which are just a renamed ZIP file. You can unzip them pretty easily. There’s a number of text files inside, plus a .JAR file under /chrome. This file is also just a ZIP file, and you can unzip it to grab the plugin code.

The quality of code obviously varies a lot, since these tend to be built by third-party developers. I’ve learned a lot of things by poking around inside other people’s extensions though. If you’re just starting out building extensions (as I am), its a great place to learn.

I’ll keep posting these as I find them. My next step once I’m done with Firefox is to build the same functionality for Internet Explorer. More learning expected there too.

Recently made the switch from Drupal 6 to 7 (I’ll probably write some technical posts about that), started playing around with Drupal’s REST services module (useful, but poorly documented – I’ll write about that too!), and am in the process of putting together a Firefox toolbar for a project (hopefully released to the public soon).

The toolbar will be heavily integrated with a Drupal website, including login control. That was an interesting challenge, since Drupal uses cookies to manage sessions, and javascript (at least in modern browsers) won’t allow cross-site cookie access. That could be a topic for another post.

In addition, I’ve been playing around with HTML5 canvas, and particularly manipulating images on the fly. I’ve experimented with it before, over the past two years, but until recently it wasn’t supported by enough browsers for me to actually use it for a live project, and in addition the javascript libraries weren’t there yet, so everything had to be hand-coded. jQuery makes things a lot easier, of course. There’s a number of tutorials on the topic now, but I find that they don’t address some of the “obvious” beginner stuff that I had to figure out through trial and error. I may write about this as well, although it will depend on the timing of a project that I can’t talk about yet.

Stay tuned for some techy material!

My immediate response is “what’s your exit plan?”.

Usually this is met with a blank stare, which is when I follow my question up with “what I mean is, how do I sell those shares, and when?”. At this point, in 90% of cases, the other party is already starting to look panicky. I then usually politely excuse myself and leave.

It isn’t that I’m opposed in general to holding equity in a project that I’m working on. Far from it. The lack of an exit plan, however, implies a number of things about the person doing the asking though:

• No business plan (if they had one, they would probably have an exit plan)
• No cash (this makes launching a business an uphill battle from the start – I know this from bitter personal experience)
• No idea of valuation (and in turn probable lack of general business know-how)
• Possible utter lack of respect for the developer (more on this bel0w)

By asking a developer to work – possibly for months, or years even – without cash, the person isn’t paying much attention to how the developer will pay their bills in the interim, and how the developer will ever get paid for the project (i.e. by selling their shares).

What they’re saying is that they want the developer to assume all of the project risk, in exchange – maybe – for some pieces of probably worthless paper. Even worse, if things go completely pear-shaped, the developer might even wind up on the hook for company debts or legal issues. Even with paper, the developer can also still wind up with their equity diluted or out and out taken away – contracts can be tricky things.

Aside from all of the above, they also clearly haven’t thought through what happens when the developer runs out of cash (i.e. they leave, or they become unmotivated).

The converse to this situation is one in which a business has a clearly defined plan, cash on hand to pay contracts or salaries, and wishes to align staff with the overall goal – this is the only time when I would ever want to be holding equity in somebody else’s company.

In addition to describing some of the mechanisms that would allow such a market to operate, I am also calling for a) the establishment of a foundation or industry association to ensure that standards are created for the necessary systems, and b) the voluntary acceptance of some level of regulation (i.e. government) by the virtual market community. I’ll make cases for both below.

I’m interested in hearing feedback. If there’s sufficient interest I’ll set up a wiki and try to get a community together.

Please note that I haven’t yet put in links and footnotes – this will be revised periodically as I find time to put that info in.

1. Overall description and necessity

There are existing efforts to build a variety of different kinds of web-based virtual markets, including currency exchanges, equities (i.e. stocks) and options.

These markets are frequently highly illiquid, poorly constructed (at a technical level, from a transparency standpoint, and at a marketing level), and suffer from extreme levels of volatility. The recent “virtual correction” in the exchange rate for BitCoins is a case in point.

The original efforts to build virtual currencies often were based on a philosophical view that decentralization and transparency would resolve many of the issues in “real world” markets; this has been undermined by the ongoing necessity of centralized markets.

There are existing plans to decentralize virtual currency exchanges – the following is an attempt to generalize this to any kind of virtual market.

2. Digital wallets, equity encoding and price information

Every user of a decentralized exchange will need to have a form of digital wallet to store information about their own holdings.

This would work in a similar fashion to a number of existing digital wallets (one example being BitCoin), but would, in addition, need to have logic for negotiating the exchange process with various intermediary parties.

It will also need to have user-adjustable privacy levels to allow for optional reporting of trade information (or even existing holdings) to the rest of the network.

As well, the wallet will need to be able to handle a wide variety of types of equity, including some complex items such as options.

The wallet may thus have fairly significant data storage requirements, particularly if it is storing information about “nearby” trades.

The process of backing up wallet information should also be far more transparent to end users than existing digital wallets. If this market is to be successful, the volume of trades and amount of total equity will be very large – too big to allow for the failure of a single computer. Information either needs to be replicated among wallets in a peer-to-peer manner, or there will need to be a role for third-party back-up services on a commoditized basis.

3. Standardized escrows, trust metrics and commissions

In “real world” markets, the broker is a key part of the exchange mechanism. They serve a variety of necessary intermediary roles in order for buy/sell operations to occur, including the process of actually negotiating the sales, the clearing process, and enforcing exchange rules such are margins.

In a peer-to-peer market, the role of a broker is taken on by escrow services. Without an escrow, there is a serious trust deficiency, in the sense that a buyer and seller need to have a high degree of confidence that a sale will conclude normally, without any way of knowing who the buyer and seller(s) (i.e. for dark pools to operate) are. If the peer-to-peer market is to operate anonymously, there is no easy way for a trust metric to be established for the individual traders. Instead, a standardized method of negotiating escrow is required.

The way the escrow system will work needs to be roughly along these lines:

• There will be a standardized API for how escrows on the market will operate with software.
• Escrows essentially become a commodity service – they will publish their transaction rates in a way that can be discovered automatically.
• The market will maintain a trust metric for individual escrows, based on the completion of sales (i.e. sales that are cancelled by buyer or seller before completion should not weigh on an escrow’s trust ratio).
• An individual trader can create their own list of escrows that they do not wish to use; otherwise the process of selecting an escrow should be completely transparent, based on trust metric and the escrow’s published commission rate.
• Since there will always be at least two escrows (and in many cases far more) involved in any sale, there will need to be a standardized method of dividing up any commissions involved, and parceling them out.
• The buy/sell mechanism will be described in detail below.
• In order for complex transactions to occur (i.e. short selling, options etc), escrows may need to have minimum capital requirements, and may also wish to enforce such requirements on traders using their service – this will need to be part of the automated negotiation process.
• It isn’t clear at all how commodity trades would operate on this type of system – certainly there would need to be a mechanism for the delivery of commodities, and this might be fulfilled by escrow operators.

The distributed buy/sell mechanism will work something like this:

• A trader places an order (buy or sell) within their wallet
• The peer-to-peer software will try to find matching orders
• If an exact match is not found, a series of partial matches may need to be made through one or more discovery services, dark pool operators or market makers (these roles will need to be standardized through APIs, and may or may not be run by different organizations; they will, however, be designed to be commoditized, in the sense that there will be many of them that are essentially identical from the standpoint of the trader).
• Once the matches are found, the trader’s software will negotiate a contract with an escrow; each of the traders in the transaction will separately do so with their own escrows. The information on which escrows are involved will be exchanged through the system.
• The contracts with the escrows will also include privacy setting information from the wallets – i.e. some users may not allow the publication of the final price, or even the fact that a trade occurred, whereas others may not mind.
• The escrows will then negotiate commission splits between each other, and handle the collection of the relevant equities, their exchange, and their return to the relevant party’s wallets.
• The wallets would then publish updates to the escrows’ trust metrics.
• This process will need to be tweaked in order to ensure anonymity, and ease of order matching.

4. Price discovery

As mentioned above, there are roles in this market for operators of dark pools, market makers and price discovery services. It isn’t clear to what extent it is possible for these roles to be completely decentralized.

In a peer-to-peer market, information about completed trades would generally be stored in the wallets of the traders, who may not necessarily be online at any given time, and who may wish to use different privacy levels in terms of sharing information. There needs to be some type of mechanism for sharing information about pricing, and this may take the role of multiple organizations that collect, collate and publish this information (possibly for a small transactional price). It is also clear that some of the information that is available in “real world” markets may not exist in a decentralized one – some trades may not be reported due to privacy settings etc. There needs to be some discussion regarding how price discovery would operate in this situation.

5. Regulation

Online, virtual markets are coming, whether we (or governments) like it or not. There’s a good chance that they will be decentralized in the way I am describing, if for no other reason than to minimize outside interference and reduce transaction costs. It is absolutely clear that ground rules need to be set, and if the technical community doesn’t do it, governments will – even if their only resort is end-user prosecution.

In “real world” markets, government regulation serves to prevent abuses such as insider trading, front running, parking and the like. While some of those kinds of abuses are not feasible in a virtual market (i.e. a broker would probably not be able to front run a peer-to-peer market order), and it would be almost impossible to force-ably regulate a decentralized market, I believe that there are several reasons why such regulation should be voluntarily accepted:

• In order for virtual markets to be successful, there will ultimately be a need for high quality equities. The only BitCoin stock exchange that I know of right now currently has two trading equities, both with market capitalization under $1000. It isn’t clear to me why any sensible investor would want to trade on such stocks; chances are that all of the current trades are by insiders of one stripe or another. A large cap company simply will not trade on such a market, since it has to operate in a number of countries that will penalize it for doing so. By regulating virtual markets, we open the doors for such equities to be traded.
• The existing virtual markets are extremely opaque; without some way to prevent insider trading, there is no way that investors (as opposed to small-scale speculators) will trade. These markets will not be successful until they can attract larger investors. In addition, many institutional investors will not be allowed by their own internal policies to trade on unregulated markets.
• By voluntarily accepting sensible regulation, we avoid the situation where regulation is set by those who don’t necessarily understand how these markets work.

It isn’t clear who the governing body for such regulation would be. Virtual markets operate in every country, and around the clock. As a result, enforcement will require participation of many polities, including many with little knowledge or understanding of how virtual markets work.

I believe that the first step will be the creation of an industry group or not-for-profit foundation that will a) assist in establishing guidelines, b) coordinate regulation with relevant polities, and c) create standards for the underlying technical specifications. The result of this process would be a set of ISO-like standards for interoperability, as well as mechanisms for self-policing (i.e. via trust metrics, or via cooperation with law enforcement).

The establishment of such a foundation will also be critical for the following reason: the hobbyist BitCoin speculator of today may be comfortable dealing with an anarchist with a nom-de-plume. Large institutional investors (and the US Congress) will not. A democratically elected foundation board, that includes people in suits (as well as bearded techies), will be needed in order to communicate with such parties.

The amount of press that this topic is currently getting highlights the fact that this is potentially the “ground floor” of a “once in a century” opportunity to build an effective replacement for existing markets. Whatever is done now will be with us for a very long time, and we need to make sure we do things properly.

6. Creation of equities – IPOs, standard contracts etc

There’s been a number of articles (note to self – I need to find them and provide links) about how equities in a digital wallet would be encoded. Generally speaking, we’re talking about some form of public key encryption that would be signed by the party that created it in the first place. The process of creating signed equity certificates would be similar to the “stock printing” process in “real world” markets, although it would likely be done directly by whomever issues the equity. There will need to be a well defined standard for how this will operate.

Standards for contracts will need to be established, and that process will also need to be determined. Currently, standards are set by individual exchanges (i.e. CBOT has a standard coffee contract that all traders on that exchange understand). Without standard contracts, each trade becomes a lengthy negotiation process (i.e. how many coffee beans are we trading, what is their quality, and where will delivery take place?).

The process for moving existing “real world” equities into a peer-to-peer market is going to be tricky, but necessary for its ultimate success. At some point, the involvement of existing organizations such as the OCC will be required.

7. The open source, peer-to-peer market

As I’ve stated above, virtual, decentralized markets are coming whether we like it or not, and there needs to be effort to do things right, up front, in order to avoid serious issues later on.

Ideally, we are talking about a self-regulating market, based on well defined, transparent, open source standards, with as few centralized “choke points” as possible. The market would consist of a large number of traders with digital wallets that store their equity positions, and a peer-to-peer mechanism for them to trade with each other. Escrow “brokers” and trust mechanisms, in addition to standard APIs and contracts will be needed in order for this market to work smoothly.

There are roles for many of the same players as in existing “real world” markets, and regulation and transparency will be required in order to allow them to do so. A guiding not-for-profit foundation, with a democratically elected board will be helpful in negotiating with existing governments and regulatory bodies. The participation of existing market players will be needed in order for this virtual market to operate – if we try to exclude them, we will only wind up creating similar (but less experienced and less transparent) roles on our own (this is already happening!); we might as well try to include them in the process deliberately.

I recently read on Slashdot that a BitCoin miner (somebody who runs a number of servers in order to – hopefully – generate sufficient new BitCoins to pay for the electricity costs) had the police come to his door. The electricity company, seeing a massive spike in his consumption, thought that he was running an illegal grow-op.

It occurred to me that the fundamental nature of a BitCoin – its value is derived from the difficulty in creating the hard encryption that underlies it – is actually a strong incentive for people to do a number of possibly counterproductive things – waste electricity, use computers solely for the purpose of creating BitCoins etc. Its only a matter of time before somebody creates malware that uses other people’s computers for this purpose. This is really no different from any other form of currency, fiat currencies included. Its just slightly more encapsulated in specific areas.

So why not create some other virtual currencies that incentivate other things. One example that occurred to me – probably not original, and definitely not scalable – would be a currency that has its value derive from live music. The medium of exchange would be some type of concert tickets, and there would be some kind of market mechanism to address the relative “value” of different musicians.

The idea would be to create a large number of different currencies of this kind, with exchanges to provide liquidity. Another extension would be to create index currencies, that derive their value from a basket of other virtual currencies. This could be used to create all kinds of interesting social incentives.

The first phase of the internet involved taking real world information, and moving it into a digital, connected format – i.e. making web pages.

The second phase of the internet involved taking that newly minted digital stuff, and bringing humanity into the picture (i.e. web pages that are “social”).

The third phase of the internet will involve taking “stuff” that was originally digital, and making it “live” in the real world. All that mobile phone geo-location stuff is just a tiny (and honestly, not very interesting) part of that.

Addendum:

The fourth phase of the internet is already upon us as well, and interestingly enough its as much about hardware as software. This phase involves breaking the physical constraints of the internet and allowing it to work seamlessly through ad-hoc, peer-to-peer, wireless networks (i.e. there’s no ISP and no phone company involved, except maybe for the long lines). This also involves replacing TCP/IP with DTN – especially if humanity is going to do anything useful in the rest of the solar system.

Flickr Open Commons - cacophonyx

The most extreme example I can think of is our calendar, where the way things are organized are based on mystical numerology from the time of the Babylonians (or maybe even earlier).

Think about it: why is the day divided into 24 hours of 60 minutes each? A metric system would greatly simplify any number of operations that need to divide time up into parts.

Modifying the way that we organize time, however, would involve replacing every clock in use around the world, and would have ramifications that simply can’t be predicted – think about all of the software that makes decisions based on specific times of day. Not just alarm clocks, but also scheduling software that involves large moving pieces of equipment. The last time a major calendrical change was made, back in the 16th century, it caused riots – and that was before modern automation technology (and things like futures contracts).

Last year I built a relatively high-traffic e-commerce website, and designed the underlying data structure for invoicing in a specific way (based on the functional specs, mind you) – and it turns out that the users of the site actually would prefer to do things somewhat differently. Fixing the problem now means performing surgery on a piece of software that runs 24-7 (yes, we have various test sites), and potentially migrating hundreds of thousands of invoices to a new (and likely incompatible) data format.

Another personal example is a quick and dirty reporting tool that I built back in the dot.com era for an insurance company. The tool was designed for a single user, and was built using Microsoft Access 2000 (and later ported to 2003). This made sense because it was never intended as anything more than a set of simple reports for one specific manager.

Of course once the tool was done, several other managers asked for access to it in order to put in reports of their own, resulting in a small modification to make the tool more compatible with multiple users, and a few more reports.

Nearly a decade later, the tool is still in use, is still growing in scope, and has now made it difficult (maybe impossible) for the entire company to upgrade their application software. Replacing the tool at this point would imply a complete redevelopment from the ground up, including replacing a big chunk of their infrastructure.

The moral of the story is simple (but sometimes hard to put into practice – as illustrated above): when building something, be aware that the decisions you make can and will have follow-on effects down the road – and try to make sure that you understand what those effects will be.

Risk Factory - by kyz - flickr.com Creative Commons

I’m interested to hear feedback regarding how other development companies measure project risk.

We currently track three general classes of risk (although in a very simplistic way) for a project:

1) Technical risk – how likely is it that we will run into something that we don’t know how to solve (or that can’t be solved as stated – or is generally insoluble).

2) Bottom line risk – how likely is it that the project will cost too much to build (i.e. it won’t be profitable). Note that even projects that are not fixed cost (i.e. are billed on an hourly or some other type of flexible basis) can run into issues if they start to cost more than some unstated budget on the customer’s end. This type of risk is frequently the largest concern on our end of things, because (like many service organizations) our largest expense is staffing.

3) Customer risk – I’ve had customers go out of business, vanish, fire us etc in the past. There are frequently warning signs from the start that a particular customer may be more risky than usual. We’ve started tracking issues in a database to try to become more adept at evaluating this sort of risk.

How does your company measure and evaluate risk? Are there relevant categories missing from my list (VaR, Black Scholes etc aren’t really relevant to software development – I think).

Recently, my company built an online “live auction” website for a customer, a project which posed a number of interesting challenges and learning experiences (the hard way, of course) regarding how to build a site that has heavy traffic. In this case, the nature of the website requires that all users see information that is current and accurate – resulting in a need for AJAX calls that run repeatedly on a per second basis per user. This project is the first one that I have worked on that required serious optimization work; typically even the heaviest custom development that my team works on is primarily focused on business use cases rather than things like speed or algorithm design; not so here.

The “coming soon” page, long before the site was launched, already received several hundred unique visitors per day (based on Google Analytics). The site launched with more than 500 registered users (pre-registration via the coming soon page), and traffic spiked heavily following launch. The initial traffic spike actually forced the site to close for several days, in order for our team to rework code. The re-launch was preceded by several Beta tests that involved registered users. Bear in mind that a registered user on most sites isn’t individually responsible for much server load. On this particular site, each user is receiving at least one update per second, each of which may involve multiple database calls.

The following is a description of some of the issues we encountered, and how they were addressed or mitigated. In some cases, work is ongoing, in order to adapt to continued growth. In many cases, the challenges that we encountered forced me to revise some assumptions I had held about how to approach traffic. Hopefully the following lessons will save a few people the weeks of sleep deprivation that I went through in order to learn them.

Project Description:

• Penny Auction website
• Technology: PHP (Zend Framework), Javascript
• Server: Various VPS packages (so far)
• Description of traffic: All users receive one data update per second; there are additional data updates every 3 seconds, and once per minute.

1. Don’t Rely Too Much On Your Server

Many web developers build code that simply assumes that the server will work properly. The problem is that under heavy load, it isn’t at all uncommon for servers to actually not function in the way that might be expected. Examples include things like file resources dropping, database calls dropping – sometimes without intelligible error codes, and even things like system time being unreliable. The following are a couple of specific examples we encountered:

a) PHP time() – When developing in PHP, it is very common to rely on function calls such as time() (to obtain system time, in UNIX timestamp form) for algorithms to work properly. Our setup involved a VPS with multiple CPUs dedicated to our use, and the ability to “burst” to more CPUs as needed. As it turned out, whenever our server went into burst mode, the additional CPUs reported different system times than “our” CPUs did. This is probably an issue with the underlying VPS software, but we didn’t have the luxury of investigating fully. This meant that rows were frequently (as in: about one quarter of the time) saved in the wrong order into the database, which is a serious issue for an auction website! When possible, use a timestamp within SQL code (i.e. MySQL’s TIMESTAMP() function) instead. Fixing the system time on the other VPS partitions wasn’t feasible, since they “belonged” to a different customer.

b) Not every database call will work. Under heavy load, it isn’t at all unusual for a SQL insert or update statement to be dropped. Unless your code is designed to check error statements, and handle retries properly, your site will not work.

2. Pick Your Hosting Company Wisely

We launched the project on one of our hosting company’s smaller VPS packages. We quickly went to one of the middle-range packages, discovered it was also insufficient, and then switched to the largest package that they offer.

In the process, we also entered a number of second tier or higher tickets into their system, including serious operating system level problems.

Luckily, we chose a hosting company that responds quickly to issues, and whose staff are familiar with the types of issues we encountered.

This isn’t something to take for granted. Not every hosting company has the ability to quickly and seamlessly transition a site through different packages on different servers, nor do they necessarily have tier 3 support staff who can address unusual support requests.

In this case, our conversations with the company seem to indicate that they have never seen a new site with this level of load in the past; they still worked valiantly to assist us in keeping things running.

3. Shared Hosting, VPS, Dedicated, Cloud Hosting?

In our previous experience, when a hosting company sells somebody a dedicated server, the notion is that the customer knows what they are doing, and can handle most issues. This occurs even where a SLA (service level agreement) is in place, and can seriously effect response time for trouble tickets.

As a result, our first inclination was to use a VPS service. Our decision was further supported by the level of backup provided by default with VPS packages at our chosen vendor. A similar backup service on a dedicated server of equivalent specifications appeared to be much more expensive.

One of the larger competitors of our customer’s site currently runs under a cloud hosting system. We are continuing to look at a variety of “grid” and cloud hosting options; the main issue is that it is extremely hard to estimate the monthly costs involved in cloud hosting, without having a good handle on how much traffic a site will receive. It isn’t unusual for hosting costs to scale in such a way as to make an otherwise profitable site lose money. That said, we will likely have to transition over to a cloud hosting service of some kind at some point in time.

4. Database Keys Are Your Friend

At one point, we managed to reduce server load from > 100% load, down to around 20%, by adding three keys into the database. This is easy for many web developers to overlook (yes I know, serious “desktop” application developers are used to thinking of this stuff).

5. Zend Framework is Good For Business Logic – But It Isn’t Fast

We initially built the entire code base using Zend Framework 1.10. Using Zend helped build the site in a lot less time than it would otherwise have taken, and it also allows for an extremely maintainable and robust code base. It isn’t particularly fast, however, since there’s significant overhead involved in everything it does.

After some experimentation, we removed any code that supported AJAX calls from Zend, and placed it into a set of “gateway” scripts that were optimized for speed. By building most of the application in Zend, and moving specific pieces of code that need to run quickly out of it, we found a compromise that appears to work – for now.

The next step appears to be to build some kind of compiled daemon to handle requests that need speed.

6. Javascript

Our mandate was to support several of the more common browsers currently in use (mid-2010), including Firefox, IE7-9, Opera, and – if feasible – Safari.

The site is extremely Javascript-intense in nature, although the scripting itself isn’t particularly complex.

We used Jquery as the basis for much of the coding, and then created custom code on top of this. Using a library – while not a magic solution in itself – makes cross-browser support much, much easier. We’re not very picky / particular about specific libraries, but have used Jquery on a number of projects in the past couple of years, to generally good results.

Specific issues encountered included IE’s tendancy to cache AJAX posts, which had to be resolved by tacking a randomized variable onto resources; this, unfortunately, doesn’t “play nice” with Google Speedtest (see below).

We also had a serious issue with scripts that do animated transitions, which resulted in excessive client-side load (and thus poor perceived responsiveness) in addition to intermittantly causing Javascript errors in IE.

Javascript debugging in IE isn’t easy at the best of times, and is made more complex by our usage of minify (see below) to compress script size. One tool that occasionally helped was FireBug Lite, which essentially simulates Firefox’s Firebug plugin in other browsers (but which also sometimes can change the behaviour of the scripts being debugged). The underlying issue is that IE does a poor job of pointing coders to exactly where a script crashed, and the error messages tend to be unhelpful. The debugging method in IE basically boils down to a) downloading a copy of the minified resource in the form that the browser sees it, b) using an editor with good row/column reporting (I often use Notepad++) to track down roughly where the error occurs, and c) put in debug statements randomly to try and isolate the problem. After working with Firebug for a while, this is an unpleasant chore.

7. Testing Server

Long before your site launches, set up a separate testing server with as close to a duplicate of the live environment as possible. Keep the code current (we usually try to use SVN along with some batch scripts to allow quick updating), and test EVERY change on the test site before pushing the code over to the live server. Simple, but frequently overlooked (I’m personally guilty on occasion).

8. CSS

Designers and web developers often think of CSS purely in terms of cross-browser compatibility. Building sites that actually work in major browsers goes without saying, and based on personal experience, CSS issues can lead to a lot of customer support calls (“help, the button is missing”) that could be easily avoided. In the case of this specific project, we actually had to remove or partially degrade some CSS-related features, in order to provide for a more uniform experience across browsers. Attempting to simulate CSS3 functionality using Javascript is not a solution for a heavy-traffic, speed-intensive site; we tried this, and in many cases had to remove the code due to poor performance.

An often overlooked CSS issue (which Google and Yahoo have started plugging – see below) has to do with render speed. Browsers view documents essentially like a multi-dimensional array of elements, and specifying elements in an inefficient way can actually have a significant effect on the apparent page load time for users. It is well worth your while to spend some time with Google Speed Tester (or Yahoo’s competing product) in order to optimize the CSS on your site for speed.

9. Why Caching Doesn’t Always Work

Caching technology can be a very useful way of obtaining additional performance. Unfortunately, it isn’t a magic bullet, and in some cases (i.e. our project specifically), it can not only hurt performance – but can actually make a site unreliable.

High traffic websites tend to fall into one of two categories:

On the one hand, there are sites such as Facebook, whose business model is largely based on advertising; what this means is that if user data isn’t completely, totally current and accurate, it is at most an annoyance (“where’s that photo I just uploaded?”). Facebook famously uses a modified version of memcached to handle much of their data, and this kind of caching is probably the only way they can (profitably) serve half a billion customers.

On the other hand, financial types of websites (think of your bank’s online portal, or a stock trading site) have business models that pertain directly to the user’s pocketbook. This means that – no matter how many users are available, or the volume of data – the information shown on the screen has to be both accurate and timely. You would not want to login to your bank’s site and see an inaccurate account balance, right? In many cases, sites of this nature use a very different type of architecture to “social media” sites. Some banks actually have supercomputers running their websites in order to accomodate this.

Underlying the dichotomy above, is the fundamental notion of what caching is all about – “write infrequently, view often”. Caches work best in situations where there are far fewer updates to data than views.

The initial version of our code actually implemented memcached, in an attempt to try to reduce the number of (relatively expensive) database calls. The problem is that our underlying data changes so rapidly (many times per second, for a relatively small number of resources, that are actively being viewed and changed by many users), that caching the data was happening extremely frequently. The result in practice was that some users were seeing out of date cached data, at least some of the time. Abandoning caching in our specific case helped resolve these issues.

10. Speed Optimization

We used Google Speed Test in order to optimize our project. There is a similar competing product from Yahoo as well. These tools provide a wealth of information about how to make websites load faster – in many cases significantly faster.

Among the many changes that we made to the site, based on the information from the tester, were the following:

a) Use minify to combine and compress Javascript and CSS files. No kidding – this works. Not only that, but if you have a large number of CSS files that are loaded in each page, you can run into odd (and very hard to trace) problems in IE, which appears to only be able to handle approximately 30 external CSS files on a page. Compressing and combining these files using minify and/or yui can save you more than bandwidth.

b) Use sprites to combine images into large files. This does not work well in some cases (i.e. some kinds of buttons), but this technique can save precious seconds of load time. We used a Firefox plugin called Spriteme to automate this task, although we didn’t follow all of its suggestions.

c) Validate your HTML. Again, another “no brainer”. The load time saved by having valid HTML will actually surprise many readers. The process of validation is a nuisance, particularly if your site serves up dynamic, user-contributed content. Set aside a few hours for this process, and just do it though. It makes a difference.

11. Don’t Forget Algorithms 101

I took several courses on algorithm design at university, and then did nothing with that knowledge for more than a decade. Surprise, surprise – a complex, multi-user site actually needs proper thought in this regard.

One example from our experience – the data that tracks the status of an auction (i.e. whether it is currently running, paused, won etc etc) can be “touched” by 9 different pieces of code in the site, including “gateway” code that responds to users, and background tasks.

It took significant effort to build a reliable algorithm that can determine when an auction has actually ended, and the task was complicated by the fact that some of the code runs relatively slowly, and it is quite possible for another operation to attempt to modify the underlying data while the first task is still operating. Furthermore, “locking” in this case may have negative ramifications for user experience, since we did not want to unduly reject or delay incoming “bids” from users.

Conclusions

1. It is very hard to plan ahead of time for growth in a web environment. Sometimes steps taken specifically to try and address traffic (i.e. caching in our case) can actually be detrimental. The process of adapting to growth can actually involve a surprising amount of trial and error experimentation.
2. Using frameworks can be very helpful for writing maintainable code. Unfortunately its sometimes necessary to work around them, when specific optimization is needed. Proper documentation and comments can help – I try to write as if I’m explaining to somebody really dumb, years in the future, what is going on in my code – and then I’m often surprised when I need my own comments later on…
3. Work with the right people. Not just your internal team, but also your hosting company etc. This can make a big difference when you are under pressure.
4. Prepare yourself for periods of high stress. Not much you can do about this, unfortunately. In most cases, it will be unlikely that you will actually have access to the resources you really need to get the job done. Make sure you schedule breaks too. Its hard. Burnout is much harder though.

We are starting a new series called Drupal hacks and solutions. This is going to be a useful collection of how-to’s and code examples, based on our experiences.

I am sure every Drupal developer out there has his own ways of styling Drupal (which as http://bomshteyn.com/2010/01/03/should-you-use-wordpress-or-drupal/ discussed is not always an easy task), we are just going to write up the cases that we encountered and solutions we used.

As always there is an easy way to do something and a proper Drupal way of doing the exact same thing. Whenever possible we try to do it the Drupal way, and would advise you to do the same, but i have to admit thats not always the case, and i would sometimes insert a SQL query into a template instead of properly using a drupal function of some sort.

It’s almost a year now since I started working on our first Drupal site; over time I have learned, and continue to learn, new “proper” ways of programming in Drupal and working “with it” versus “around it”.

As you work on a project you have a problem and find a solution, but by the time you get around to the next project and encounter a similar or even same problem (the latest project we were working on was approximately 400 of programming hours) you no longer remember how you solved it last time. In theory you could just open the last project’s code and find the solution, but sometimes its easier said then done, first you have to remember in which project you had this issue and second since an issue in Drupal could be fixed on so many levels (core, theme, modules….), where to look becomes an issue by-itself.

So here we are faced with a case of our own that needs a solution.

Our first urge was to create an internal wiki where we would write down this type of stuff for future use – knowing first hand how hard it is to find an up-to-date solution to some very common problems in theming Drupal. Plus the fact that we always wanted to give back to the open source community of Drupal. We decided to do it as a blog category on our site so that it would be searchable – not just internally but by the whole ever growing Drupal community. As a starting point we will give you an example post, so that you know what to expect.

How to add a custom login box into your Drupal theme

You will probably want to show this only for non logged in users so here is an example of the code you would use:

<?php
if($logged_in){?>
// put some code here
<?php } else {?>
<?php  global $user; ?>
<form action=”<?php print $front_page.’user/login/?’.drupal_get_destination();?>” method=”post” id=”user-login”>
<label for=”edit-name”>username</label>
<input type=”text” name=”name” id=”edit-name” value=”" tabindex=”1″/>
<label for=”edit-pass”>password</label>
<input type=”password” name=”pass” id=”edit-pass” tabindex=”2″/>
<input type=”hidden” name=”form_id” id=”edit-user-login” value=”user_login” />
<input type=”submit” name=”op” id=”edit-submit” tabindex=”3″ value=”Submit” alt=”Submit” />
<p>Forgot <a href=”/user/password”>password</a>? &nbsp; Or <a href=”/user/register”>Create New Account</a></p>
</form>
<?php }?>