The following blog entry describes the solution to an issue encountered by Jeremy and Nathan in moving data from an old website to a new one. Based on a brief search for a solution, it looks like many other people may find a use for the (admittedly rather crude) code that we wrote to solve the problem. We hope this helps!

We’ve recently had to convert data from an old custom-built shopping cart, to a new website based on Drupal and Ubercart. The old website had approximately 5000 products, with a data structure that was completely unlike the one used by Drupal to store data. We initially tried using the Drupal data loader module, but as other people have discovered, it isn’t necessarily a good fit for loading Ubercart product data, particularly when the products have images, or – and this is especially the case – custom data fields.

You can download a copy of our script here.

In the end, we put together a custom php script to pull data directly from one database to another. The script is fairly crude, but it may be useful to somebody trying to import data into Ubercart from another non-standard system.

As a result, I will link a copy of the script below. Please bear in mind that a) we accept no responsibility  whatsoever for its use (i.e. back up your databases first!!!), and b) while we have documented our assumptions, this is a crude script that is specifically designed for our particular needs. Be that as it may, you may be able to find a use for this.

Additionally, it was a pain just getting a list of the Drupal and Ubertcart tables that needed to be looked at in order to do the data conversion. The list below may be useful to you as well.

How it works:

The old shopping cart had two relevant tables: a category listing (hierarchical), and a products table, which included some customer specific fields, as well as a single url to a product photo.

Drupal/Ubercart, on the other hand, uses a large number of tables to represent the same data. This is because products use the same node-based system as the rest of Drupal, in addition to a number of product specific tables.

In addition, product images are dealt with using Drupal’s file cacheing system, which is relatively complex in nature. All in all, our data goes from two tables to approximately nine (depending on your specific needs) in Drupal.

The script has some handling – very buggy still – in place for keeping track of the current position in the old database. This means that you can run the data conversion in small chunks, checking at each point to see how well it is doing. You can also set how many records to convert each time. In order for this to work, a very simple table to track the record pointer needs to be added into the old database.

We did not cover the conversion of product categories, since typically there are a much smaller number of categories than products, and in addition, the Drupal data loader does a reasonably good job of loading them.

List of tables involved:

• node – in Drupal, everything is a node, including products. Each new product record will need a corresponding node of the correct type.
• node_revisions – this table contains much of the actual textual data (i.e. product description text) involved in displaying the node.
• content_type_product – if you need to create custom fields for your Ubercart products, the data will go here. Our code in this section will probably not map directly to what you’re working on.
• uc_product – where the product records “live”. There will be one record here for each of the product nodes.
• uc_product_stock – contains stock-related data (i.e. how many of the product you have on hand etc) for products.
• files – each product image needs to have a record in this table, which stores the file names and locations – but does not map these images directly to the product records (you need another table for that!).
• content_field_image_cache – this is the mechanism that ties a product image to the underlying product. We didn’t really try to work with Drupal’s actual cache mechanism, so after you have loaded the products, you may want to clear the cache.
• term_data – this is where product category information resides, based on Drupal’s vocabulary / hierarchy system. We hacked the solution in a fairly inelegant way – by doing a category name lookup to retrieve the term id.
• term_node – a connection table between a term (i.e. a product category) and a node (in this case the products).

Base assumptions:

• We assumed that the old and new websites both have mysql databases.
• Both databases will probably have to be on the same server (or at least one of them needs to be externally accessible if that isn’t the case)
• The script should be uploaded and run from the new website’s location.
• We assumed that there is only one product image per product – although this would be fairly easy to modify.
• We manually added our categories to the new website
• We manually moved product images across – there are comments in the script where you could automate this though.
• Error checking is extremely rudimentary; if this scripts hits a problem it just stops.
• If you hit data conv snags, the fastest way to proceed is to restore your backup database, reset the pointer record, and start over.

I also once worked with somebody who would regularly call into the office with the complaint that their GPS had succeeded in making them lose their way.

The story reported on Yahoo’s tech blog here (http://bit.ly/lEXUH) is a pretty good one though.

Almost as good as the guy that went off the autobahn, across a field, and smashed his 7-series BMW into a porta-potty.

QR Tag

The odd looking image here isn’t a Rorschach blot.

Its actually a QR code, something that is old news in Japan, but never quite managed to take off in North America.

In this particular case, assuming you had a QR code reader (in Japan apparently most cell phones do), the image encodes a link back to this website.

If you want one of your own, you can go here to get one: http://www.qrtag.net.

Now I just need to figure out how to get my phone to read them…

For those of you who aren’t familiar with it, I suggest taking a look at Grader.com, a tool provided by web marketing gurus HubSpot.

Their website grading tool provides a host of useful information that can help you fine-tune your site.

I’ve been playing around with their tools for the past few months, and they’ve been extraordinarily useful in terms of tweaking things to make them more search engine friendly. Also useful is their Twitter profile grader.

While assisting one of my staff with a problematic installation of some open source software on a server, I decided to clean up certain files that we no longer needed with “rm -r *”.

Only to discover that I was in the wrong directory.

It didn’t help that we had been working directly on that computer for several days.

I was saved from losing a week’s work only by the fact that the backup from the morning was good (you never know with backups). We still lost a few hours of work, but that’s much better than it could have been.

Moral of the story: a) backup even more regularly than you think necessary, b) keep a local copy of your working files, c) don’t use “rm -r *” unless you’ve double checked what it will do.

Image courtesy of "gutter" on Flickr. Creative Commons.

I just spent a chunk of this afternoon fixing up a friend’s website which was hacked. The hacker appears to have gained access through a decade old shopping cart (not in use, just sitting in a folder on the site), and then proceeded to insert obfuscated javascript code into every page on the site (several hundred pages, with the code slighly different on each page).

This is the fifth or sixth site I’ve had to clean up in the past year or so, and its always a painful job – I’m pretty good at spotting code that shouldn’t be in a page, but with a large website it can be hard to be certain that it has been completely fixed. And there’s no guarantee that the original loophole that was exploited has been removed. Even under the best of circumstances, cleaning up this sort of mess is a painstaking process.

The following is intended for web designers who aren’t coders – but who use scripts that they have located on the web. Some intro level programmers might benefit. Experienced web programmers should go directly to the following link and do some review: http://cwe.mitre.org/top25/

1. Be very careful about downloading “free” scripts off the web. Do yourself a favour and scan the code before using it. If it has been obfuscated, or it looks odd, you probably want to avoid using it. You don’t need to be a programmer to get a feel for nefarious code.

2. When putting together a website that has any kind of dynamic functionality – be it javascript, a php script on the back end, or something else – bear in mind Jeremy’s Addendum to Murphy’s Law: Whatever can be hacked, will be hacked. There are a lot of common loopholes that hackers exploit that could be easily avoided by looking at code with a cynical eye and trying to figure out how it can hurt you.

3. Periodically review old websites that you’ve done. Code that used to be fine may no longer be so safe. Also, as you learn from mistakes, you may notice all kinds of things that are dangerous in your code.

4. Its also really worthwhile to look at the Top 25 Dangerous Bugs list, linked above. A periodic review is in order. Speaking of which, I’m adding that to my to do list.

5. Verify ALL inputs to a script. If you think you have verified them, get somebody with a cynical bent to test it. If something is up on the web, it is guaranteed that somebody will try some oddball and highly unexpected inputs just to see if they use your script for their own purposes.

6. Remember at the end of the day that there’s absolutely no such thing as a hacker-proof piece of software or hardware. Make regular backups. Assume you’re going to need them.

I just want to finish with an anecdote.

I used to operate a small hosting company along with some of my other duties at my former company.

One day, one of our servers started broadcasting vast volumes of spam email, to the point that we had to shut down the outgoing email service.

I spent a few hours reading log files, trying to pinpoint what exactly was happening. I finally narrowed it down to a script that had been uploaded a few days prior on one of the client’s accounts.

The script was basically a feeble attempt to try and implement a CMS (content management system). Basically the way it worked was that any GET input to the main script was assumed to be the name of an html fragment file, and was included into the script with no verification whatsoever.

If this means nothing to you, you’ve probably seen websites that have URLs something along these lines: index.php?id=123. The “id=123″ part can be parsed out by the script as an input. In this case the links looked like this: index.php?page=contact.html.

The script just assumed that contact.html was a piece of HTML code, and included it in.

It didn’t take long before half the hackers in the world were sending the script stuff like this: index.php?page=path_to_malware_or_spam_script. And our server was running those bits of malware as if they were located locally.

There's an opportunity to incorporate social media into the classroom. Will school boards take it?

If I had a dollar for every kid that tries to get me to answer their homework questions on Yahoo! Answers, I might have a better than average chance of paying all my bills this month.

Yes, we happen to live in an age where things are changing pretty fast. It still puzzles me that the most common reaction by schools and universities to the social media phenomenon is to try and ban it from the classroom. Hence the proliferation of websites that try to catch cheaters.

If I was running the show, I would try a different tactic: co-opt social media. Make it part of the game. There’s a great learning opportunity here, and it is being missed – at least in North America. In Europe, there’s a heavy push to incorporate e-learning into the classroom (see http://en.wikipedia.org/wiki/Computer_aided_instruction for some interesting related topics).

Here is how I react when somebody tries to get me to do their homework for them: hey kid, there is an awesome learning opportunity here. I’m not going to solve the problem for you, but I will try to teach you a few interesting things. Maybe I’ll rephrase the question for you so you can understand it better. Maybe I’ll point you in the right direction so that you can discover places online where you can learn more about the problem at hand. Maybe I’ll give you a few pointers on ways to approach a solution. Sounds more like a tutorial? Self assisted learning opportunity?

One critical factor is that one really needs something like a walled garden – at least initially, and at least for younger students. If you toss them onto Yahoo Answers and tell them “good luck kid”, they’re going to come back with some interesting (and probably odd) notions about how the world works. For one thing, many of the so-called experts on sites like these, ain’t. Even on the late, great Askme.com, there were more than a fair share of kooks. Many of the e-learning projects underway (i.e. the Second Life-based project in the UK) are building things around such walled gardens.

If schools – or maybe school districts – had a site that only kids and teachers could login to, it could be a powerful tool. You need a critical number of users before something like this becomes useful. I don’t think one school is sufficient. On the other hand, if the whole world is involved, it may become too unwieldy (and expensive to maintain – let alone the factor of who owns and manages it).

Let other kids get involved in teaching their peers. After all, teaching something is often the best way to learn it.

Let adult teachers supervise and guide the process. I envision a system that categorizes data by topic, and allows the teacher to put a filter on it – right now you can learning anything you want about math. Here’s todays quick lesson and some questions to answer. Here are the resources to learn more. Need help? Here’s what everyone else in the class is working on? Here’s who else in the school district can help you? Here’s what last year’s class did.

Put in scoring mechanisms so that students can get competitive if they want. Help your fellow student, two points. Get rated for the best question by teach and peers? Bonus points! It would be critical to balance a competitive system so that it doesn’t leave some students behind, possibly through an opt-out system. Or just let kids see their own score and rank, without access to anyone else’s.

Build in the day’s lessons in a way that the students can explore the topic in their own way and at their own pace, but with guides and video tutorials to help them if they get stuck. I know that this kind of learning methodology doesn’t work for everyone. There has to be a way to incorporate self directed learning into a pedagogical system though.

I wish there had been something like that when I was growing up. Yes, there were computers in the classroom (I got lucky with my schooling). Yes, we learned how to program in Basic and Logo. I also grew up reading Ender’s Game, and there were definite precursors to e-learning social media in there. The concepts involved here aren’t new, and the technology involved isn’t particularly challenging any more. There are even some fairly big companies building pieces of the puzzle – hence Blackbaud and their myriad competitors (e-learning overall is at least a $50 billion USD per year industry). All something like this needs is a vision, some corporate sponsors, and a lot of courage from school boards.