Performance Testing @ the Frontline

A hidden world where small things make a big difference

More about the online-ticket system …

Posted by Kim on Tuesday, September 27, 2011

It’s now been two weeks, and the on-line ticket sales system is still not working as it should. All the sales kiosks in the whole country are now closed until further notice and the system is still slow, and has serious defects.

During these two weeks there has been some more information released to the public, both from the vendor (Accenture), the customer (VR) and the hosting company Tieto.

Among the more interesting ones:

  • The system was tested by the vendor in India. (Link)
  • Tickets have been “lost” and not delivered even when money has been charged (Link)

Lets consider the first one

What’s wrong with that first one? In my opinion the VENDOR of the software must never ever, under any circumstances, be allowed to do the final testing of the software. If I where in the paying customers shoes, I would not be able to trust the vendor to do proper testing, and even if I demanded test results I would consider them even less trustworthy than the testing itself.

The fact it was tested in India means nothing in itself. The people over there are as competent as anywhere. The problem is that they are working for the vendor.

In reality the business is usually setup so that the project is budgeted beforehand, and the sales agreement probably does not include testing with more than a few words, nor much about the fixing of defects. And to top off the weird things: In some cases if the vendor makes the deal and the vendors sales people get their bonuses even before the actual system is even created, installed, or tested.

After the deal is made, a different group of people actually start to design, code and build the final system. It is often also this same group of people that are later charged with the testing of the software. The project has a “test group” to test what the “development group” have produced, and all this managed under the same “project”, and the same people responsible in the end.

The project is most probably bound by release deadlines. And remember that the bonuses are also tied to these deadlines. I do not think the project lead wants to delay the project release dates just because of a few bugs? Very often then thinking is: “We can put them on the defects list and schedule to fix them later. The important thing is to get a release done before the deadline so we get our bonuses.”

I would say that the focus and goal the the project is a little skewed here?

Also considering that a separate agreement is often in place, stating that “bugs” and “issues” in the delivered system are taken care of in the next release. Of course this “maintenance release” or “update release” is not free of charge! This setup is perhaps the most ideal setup ever to produce the worst quality software ever. In every place the focus is on the wrong thing.

How should they have done the testing then?

This is a hard question, and often both the customer and the vendor are reluctant to admit that testing is costly and takes a lot of time. Proper testing may also delay the release dates and in the worst case cause serious redesign of the whole system.

In truth, even proper testing will never be able to save a badly designed, ill implemented system after it has been done.

Proper testing however CAN prevent a bad design and prevent ill implementation.

In this specific case I would have, from the Customer side, chosen to buy the quality assurance as a whole from a third party that specializes in quality assurance. There are several of these companies around, and they have nothing to GAIN nor LOSE by doing proper testing.

The testing team (QA) should be separated from the development team (vendor) in regards to money. This will ensure that the QA team has no problem with reporting the errors they find. It should also make the developers want to produce better quality since they now have an independent testing authority. Side note: Never tie the QA teams money to the number of defects either!

And as this was a large project I would suggest several experienced quality guys (Functional, Automated and Performance) to work WITH the developers during the development, to prevent the bad designs and ill implementations before they are even done.

And what about the second point (remeber, up there the two bullet points)?

In addition to all the above there has been discovered a serious bug in the way the system handles payments and tickets.

It seems there have been several (read hundreds) of cases where people have been charged for a ticket, but have never received the ticket! I even heard about a mom that bought a family ticket twice, and was promptly charged and never received either of the tickets.

VR officially says that to get a refund for a “lost” ticket: “You must call our non-free customer service, queue for your turn and then we’ll look into it”.

Say what? Call a customer service number that COSTS you even MORE money than you have already lost, not counting the TIME spent on this, and the frustration?? Just gobsmacking how they treat their customers!

The system was thoroughly tested according to the vendors spokespersons. So what did they really test? And How? These are questions we may never get answers to, but we sure do know the results of the testing 🙂

What are the effects of all of this?

Now what are the effects of all of this, besides the obvious vendor PR?

There have been reports of the people in the customer service department of VR saying that they want more money because of the increased workload. Not to mention they have to listen to all the people calling in and shouting at them for a failed system! The customer service people are definitely not to blame for all of this.

I say that brave souls sit there and answer the phone these days.

VR’s reputation has been tarnished for a long time by this. They have never been very good at keeping the timetables, especially during winter with the French train carts, but this new scandal with the tickets really puts them in a bad light. This ultimately leads to less people using the trains, meaning less money for VR, leading to higher ticket prices …

So, from all the people who use the trains daily: Thank You for a well managed project, well built and well tested system. (NOOOT)

PS.

Did you know that a one-time 1h ticket costs 4€ in Helsinki. This gives you the option to travel with Bus, Train or Tram inside Helsinki, Espoo and Vantaa areas for free for 1h after the purchase of the ticket.

If I drive my car to the center of Helsinki an park it for 8h (workday) I pay 8€ (1€/hour).

So, WTF? 2 times 4€ with commuter traffic is THE SAME PRICE as parking my car for 8h? Which one do you think I choose to use, especially with the problems of late!

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: