Lichtman Consulting

Nathan did a nice article on tools and tricks for creating new sites in Drupal. Worth a read: http://bomshteyn.com/2009/12/29/couple-of-things-i-learned-while-creating-sites-in-drupal/

The following blog entry describes the solution to an issue encountered by Jeremy and Nathan in moving data from an old website to a new one. Based on a brief search for a solution, it looks like many other people may find a use for the (admittedly rather crude) code that we wrote to solve the problem. We hope this helps!

We’ve recently had to convert data from an old custom-built shopping cart, to a new website based on Drupal and Ubercart. The old website had approximately 5000 products, with a data structure that was completely unlike the one used by Drupal to store data. We initially tried using the Drupal data loader module, but as other people have discovered, it isn’t necessarily a good fit for loading Ubercart product data, particularly when the products have images, or – and this is especially the case – custom data fields.

You can download a copy of our script here.

In the end, we put together a custom php script to pull data directly from one database to another. The script is fairly crude, but it may be useful to somebody trying to import data into Ubercart from another non-standard system.

As a result, I will link a copy of the script below. Please bear in mind that a) we accept no responsibility  whatsoever for its use (i.e. back up your databases first!!!), and b) while we have documented our assumptions, this is a crude script that is specifically designed for our particular needs. Be that as it may, you may be able to find a use for this.

Additionally, it was a pain just getting a list of the Drupal and Ubertcart tables that needed to be looked at in order to do the data conversion. The list below may be useful to you as well.

How it works:

The old shopping cart had two relevant tables: a category listing (hierarchical), and a products table, which included some customer specific fields, as well as a single url to a product photo.

Drupal/Ubercart, on the other hand, uses a large number of tables to represent the same data. This is because products use the same node-based system as the rest of Drupal, in addition to a number of product specific tables.

In addition, product images are dealt with using Drupal’s file cacheing system, which is relatively complex in nature. All in all, our data goes from two tables to approximately nine (depending on your specific needs) in Drupal.

The script has some handling – very buggy still – in place for keeping track of the current position in the old database. This means that you can run the data conversion in small chunks, checking at each point to see how well it is doing. You can also set how many records to convert each time. In order for this to work, a very simple table to track the record pointer needs to be added into the old database.

We did not cover the conversion of product categories, since typically there are a much smaller number of categories than products, and in addition, the Drupal data loader does a reasonably good job of loading them.

List of tables involved:

• node – in Drupal, everything is a node, including products. Each new product record will need a corresponding node of the correct type.
• node_revisions – this table contains much of the actual textual data (i.e. product description text) involved in displaying the node.
• content_type_product – if you need to create custom fields for your Ubercart products, the data will go here. Our code in this section will probably not map directly to what you’re working on.
• uc_product – where the product records “live”. There will be one record here for each of the product nodes.
• uc_product_stock – contains stock-related data (i.e. how many of the product you have on hand etc) for products.
• files – each product image needs to have a record in this table, which stores the file names and locations – but does not map these images directly to the product records (you need another table for that!).
• content_field_image_cache – this is the mechanism that ties a product image to the underlying product. We didn’t really try to work with Drupal’s actual cache mechanism, so after you have loaded the products, you may want to clear the cache.
• term_data – this is where product category information resides, based on Drupal’s vocabulary / hierarchy system. We hacked the solution in a fairly inelegant way – by doing a category name lookup to retrieve the term id.
• term_node – a connection table between a term (i.e. a product category) and a node (in this case the products).

Base assumptions:

• We assumed that the old and new websites both have mysql databases.
• Both databases will probably have to be on the same server (or at least one of them needs to be externally accessible if that isn’t the case)
• The script should be uploaded and run from the new website’s location.
• We assumed that there is only one product image per product – although this would be fairly easy to modify.
• We manually added our categories to the new website
• We manually moved product images across – there are comments in the script where you could automate this though.
• Error checking is extremely rudimentary; if this scripts hits a problem it just stops.
• If you hit data conv snags, the fastest way to proceed is to restore your backup database, reset the pointer record, and start over.

I’ve finally taken Nathan’s advice and installed Disqus to handle the comments on this site. Please let me know if you notice anything broken as a result. Supposedly there are some nice features, including the ability to leave video comments.

Continued from Part 2 – http://lichtman.ca/uncategorized/whole-life-approach-to-website-development-part-2

In Part 2, we covered some of the stages during development and just after the launch of a website.

8. Operations

I’ve frequently seen websites launch successfully, only to lose steam rapidly, once the obvious difficulties in actually running a web-business come into play.

Operating a successful web-based business can be a lot of work, no matter how much effort is placed on automation.

A website that sells a product can have a major time committment required in order to fulfill orders (i.e. package them up and mail them), or to answer queries.

Content-based sites can be even harder to maintain, particularly when the general public has the ability to participate in the content creation process – large social media sites often have a significant number of staff dedicated to removing content that contravenes the terms of service.

I’ve recently worked with several customers who have partnered with fulfillment centers in order to offload a portion of the order fulfillment process onto specialists. The websites have been designed to integrate with the fulfillment center’s systems in order to keep the order statuses current. I’ll eventually post an update here when I’ve seen how well this works in practice.

Creating a plan – possibly even as early as the business plan stage – for how operations will work, the amount of time needed to keep a website running, and whether any staff are required for this purpose (and particularly how much that would cost), is an effective way of making sure that the resources will be in place to run things after launch. It also gears owners of web businesses up for the long, ongoing task of running an online business.

9. Maintenance Cycle

Image by akseabird via Flickr

Most web developers approach the ongoing maintenance of a customer’s site as a form of additional revenue after a project has completed.

Few owners of websites appear to budget for ongoing maintenance related issues – which could include bug fixing or building additional functionality – before or during development.

For developers, there’s a risk associated with the maintenance cycle, given that an ongoing series of invoices (unless well justified) could result in the customer leaving for another development company.

Ensuring that new owners of web businesses understand what is involved (particularly with regards to costing) in maintaining a website over time will go a long way towards forging a better partnership between developer and customer.

10. Creative Destruction – When to Scrap It and Start Over

I’ve worked on a number of websites where the site had some initial success, which dropped off over time. Unfortunately some products are faddish in nature, which means that there is a very well defined life cycle for the product.

The products or services sold by a website may not be the only things subject to this inherent life cycle.

A website built today may not work as well in the future. Technologies change, people’s design taste changes, and websites do become outdated over time.

As some point in time, a decision may need to be made to scrap a site and start over.

This often seems to happen on an ad-hoc basis, where a customer becomes infuriated with a developer and leaves for another web development company; the new company will typically try to sell a complete start-over, as opposed to modifying somebody else’s work.

By ensuring from the start that customers understand that there are inherent lifespans involved in a web business (just like any other), developers can potentially develop a longer term relationship with their customers that lasts beyond a single project.

Conclusions

There are a few quick conclusions that can be drawn from all of this:

• A web business involves a lot more than just throwing together a website and putting it up on the internet. If you ignore the fact that its a business like any other, you may be setting yourself up for failure.
• Web development companies are only part of the “supply chain” involved in building a successful web business. Other companies that are often involved include: business consultants, accountants, lawyers, loan companies or investors, PR people, SEO/SEM companies, supply chain management or fulfillment centres, phone support companies and many more. It makes a lot of sense for web developers to cultivate relationships with each of these kinds of companies, in order to be able to provide a full “package” to their customers.
• It takes time and extended effort to build a successful web business. Both you and your vendors need to have tenacity, because a lack of staying power won’t cut it. Despite anything you’ve seen in the news, most successful online businesses are the result of many, many years of hard work.