Minneapolis, MN
2/18/2004

What Hap? What Hope of Good?

Retail has always been slow, suspicious, and cheap when it comes to innovative technologies. But there is a compelling case for the newest wave of innovation. And if retailers buy that case, Retek could be a big beneficiary.

Will Retail Always Disappoint?

I'd like to talk about a market that has had a habit of disappointing technology innovators for the past 15 years. Retail.

You would think that retailers are technology leaders. Systems are most useful where there's lots of data and lots of activity, and retailers certainly have plenty of both. But most retailers are laggards, Wal*Mart being the big exception.

Why? Well, for one thing, retailers are cheap. I'll never forget one senior exec at a big retailer who would instantly translate any new IT cost into the number of gallons of milk he'd have to sell to pay for it. (Hint: his margin on milk was about a penny.) Needless to say, he never paid very much.

For another thing, the scale of retail systems is overwhelming. As SAP has found out to its pain, Unix systems and 4GL databases that work well in other industries just can't handle the data load in retail.

And finally, in retail, skilled manpower is in short supply. If you are a retailer and you want to try an innovation, you'll have to hire people to push it through, and they will encounter a lot of resistance from people who think the old ways are good enough. That's true in any industry, but it's worse in retail.

If glacial indifference to innovation and a reluctance to spend money were to deter software salespeople, no software would ever get sold. So even in this industry, there are vendors of large transaction systems, like Retek and JDA, that have managed to establish beachheads. Their core products are what are called merchandise management systems, systems that keep track of all the stuff that retailers buy, warehouse, and sell. In retail, merchandise management systems are the equivalent of ERP systems in other industries; so there are also best-in-breed players that sell smaller systems that do more specialized tasks.

History has never been very kind to the vendors of merchandise management systems. There was never a wave of frantic adoption, the way there was with ERP. And the venturesome retailers that did adopt have sometimes had their troubles. Implementation times have been known to be pretty long; and the implementations have not been entirely immune to failure. You would expect this. Customers that are cheap and suspicious of innovation are going to have problems putting in expensive, innovative systems.

Even today, the outlook is less good for these companies than for the ERP companies. SAP, Oracle, and PeopleSoft are profitable, and they seem to have ridden out the long downturn. Retek, by contrast, is still not quite profitable and cutting costs where it can.

For pundits, of course, the darkest hour is a good time to predict the dawn, and in this piece I'd like to do just that. It seems to me that retailers are about to "discover" some "new" technologies that are really compelling. Beneficiaries of this discovery (if it happens) will certainly be the vendors of the new technologies. But unusually, I also think that the vendors of transaction systems, especially Retek, could benefit as much or more.

The usual caveats, of course. "Soon" in retailspeak is about the same as "after I graduate" from the slacker crowd. And of course, even if demand improves in the abstract, Retek (or other potential beneficiaries) will have to anticipate it and move effectively to meet it. More on this later.

Several regular financial analyst readers of the Short Takes have asked me to be a bit crisper about where I stand with respect to companies I review. Reading over a few past issues, I can see why. So starting with this issue (and with Retek), we will be putting out a series of one page outlook sheets on individual companies. We'll call them "Short Sights." Short Sights will be available only to paid subscribers.

Applying Science to Retail

To understand what these new, compelling technologies are and why they're compelling, you need a bit of background on retailing.

In retailing, the essential decisions are what are called "merchandising" decisions. These are answers to questions like, "Which products should we carry and where?" "How much product should be on the shelves at any one time?" "What should be on an aisle?" "What should be the price for each item?" "What should we put on sale, and if we do, how much extra should we order and when?" "What should the sale price be?" Or "How can we draw customers into the store."

All retailers want to improve their merchandising. When you make a penny on a gallon of milk, you can see that it's important only to order the milk you'll need and never to knock $.10 off the price unless you're 100% positive you'll get other sales that will make up for it. But in my experience, most retailers aren't crack merchandisers. I think it's the lack of talent that gets in the way. People try, but the results are pretty hit or miss and always in need of more talent.

Let me propose a startling, radical way of improving merchandising decisions. Why not use data? After all, you have a record of what sold and what didn't. So this year, when you're planning a promotion of seasonal merchandise, why not look at what happened last year, first. If the promotion was a bust and you ended up with bags of the stuff, why not schedule this promotion a little earlier and mark down faster?

But don't do this all manually. You can't. There are too many promotions, too much data. Make the use of data as automatic as possible. Maybe you can even write a little program that figures out whether last year's promotion was a bust and prints out a recommended markdown schedule. While you're at it, let people set implement the schedule with a push of the boutton.

Heck, while you're at it, why not also write some programs that analyze sales throughout the year, at the very least, offer the category manager suggestions about ordering levels or pricing. If we wanted to dignify those programs and make them more credible, you could tell the merchandise managers that you were Applying Science to retailing.

OK, I admit it. It's not my idea. In fact, it's such an obvious idea that Wal*Mart has been trying various versions of it for at least 15 years.

So obvious that right now, today, there are application programs available commercially that can Apply Science to every one of the questions I've mentioned above. There are programs that try to manage shelf space, try to figure out which brands to carry, try to optimize pricing and promotions, try to figure out when to mark down seasonal goods, and even try to figure out how to replenish in response to anticipated demand.

Some of them have been around for 10 years.

The Progress of Science

I am not, therefore, arguing that there are brand new Science ideas that are about to take retailing by storm. There are old ideas, I'm arguing, that have become more appealing than the used to be. Why are they now more appealing? Consider the following:

• The programs available now are better. The Science surrounding these retail programs really has improved, and there are now more people who understand that science and can apply it to more different areas.
• The quality of data in retail systems is gradually improving, as things like UCCNet and WWRE/GNX gradually gain acceptance and as the spectre of RFID begins to force suppliers to clean up the product information that they provide. As data quality improves, there is less noise, and one's ability to pick out the signal improves.
• It's easier to act on the Science. The cost of broadband connections to retail stores is gradually going down, making it more reasonable for stores to integrate inventory, POS (point of sale), forecasting, and ordering systems.
• Systems are less likely to choke. As computer systems get more powerful, it becomes increasingly possible to handle the huge amounts of data that retailers generate.
• There are too many stores. Retail itself is going through some structural changes, which make it more likely that some customers will be willing to take chances.
• There is more proof. The results of pilots run by best in breed vendors at brave early adopters are beginning to come in, and many of them are positive.

All these are gradual trends; I am not saying that they will cause a stampede this year. But what they add up to is an increasing acceptance of the idea that you can Apply Science to merchandising (and pricing) problems and get some systematic benefit. It means that some of the companies that were thinking about it last year will try it this year and that companies that thought Applying Science was ridiculous last year will be putting a (small) line item into next year's budget.

Why Such a Slow Acceptance Rate?

"But wait a minute," I hear somebody saying. "Why so slow? If Science really works in retailing, why isn't every retailer lining up to adopt it? Merchandising and pricing, after all, are at the heart of retail. If you can do it significantly better than the other guy, you get a huge edge."

Well, you've hit the nail on the head. The fact is that there's a lot of justifiable skepticism about whether any of this Science works.

Some of the skepticism is there because it suddenly seems that there are so many things you can do. There's Scientific markdown optimization, Scientific promotion planning, Scientific forecasting and replenishment planning, Scientific assortment planning, Scientific space planning, even Scientific collaboration. Does all of this stuff work? Which works best? Why? Well, no one knows.

But the skepticism runs deeper than that. It really seems that people just don't believe in any data-driven application that does fairly heavy mathematical analysis and tries to take over some part of the "art" of merchandising from the people who now do it.

And this skepticism is appropriate. If you were a retailer, wouldn't you think as follows?

"Any black box presided over by some gowned statistics weenie is going to be generating hoo-ha, as far as I'm concerned, until proven otherwise, especially when my expert, who's been doing it for years, says it's nonsense. I want to understand how it works and why it works and what its weaknesses are before I surrender myself to it."

Being software vendors, the vendors of Science Applications do little to allay this skepticism. I've talked to a lot of them, and they pretty much hand you a line. Their product will transform retailing and end world hunger, they tell me, but they can't say how; "That's Ph.D. stuff." I have even had one vendor tell me with a straight face that they can predict the weather for the next selling season and make their clients a pot of money by allowing them to schedule promotions only for weekends when the weather will be dry and sunny. How? Ph.D. stuff.

Well, I have a Ph.D., and I can tell you from personal experience that when a Ph.D. waves his hands and says implausible things, he is probably on pretty shaky ground. And that's the case here.

You see, there are two fundamental problems with any of these Science Applications. First, real-world data is so messy that extracting meaningful relations from it is always hard and always partly a matter of intuition. Second, predicting future performance from past behavior is always risky, because things can change in the interim. When the World Trade Center went down, that blew every economic prediction based on past trends right out of the water.

Take markdown optimization for an example. The basic idea is to predict the price sensitivity of a seasonal product like bathing suits over the course of the spring/summer season. At the beginning of the season, a fair number of people are willing to pay full price; at the end, most expect to buy them on sale. Then use your sensitivity function to schedule the markdowns in a way that optimizes your profit at each point and still gets rid of the swimsuits.

People usually do this by looking at comparable swimsuits from last season, figuring out the price sensitivity function, then using linear programming to maximize the profit. The Ph.D.s who work out the price sensitivity function use fancy mathematical algorithms with very long names to get a read on what is significant in the data. But their application of these algorithms is by no means pure. People use their intuitions, make guesses about whether anomalies are meaningful, etc. And what they end up with is only a best guess. It's a matter of probability whether the algorithm has successfully isolated the meaningful data and interpreted it correctly and a matter of hope that the Ph.D. has improved the result rather than damaged it.

And remember, even if all that data on swimsuit sales has been analyzed successfully--the geographical segmentation worked out, the spike in sales in New York City during Fashion Week factored away, etc., etc., the real world may intervene. If this year's swimsuits are ugly, whereas last year's were very attractive, people will resist paying full price more than they did last year. And if the weather is unseasonably warm, people will be content with a lower markdown. So is your price sensitivity function right? You'll never know.

For retailers who are interested in these products, the general approach is to pilot the software by running a bake-off. A group of comparable products will be divided into two: the Ph.D.s will set the markdown schedule for one group and the specialists for the other. The results are supposed to "prove" which is better. (Obviously they don't really prove anything, any more than the clinical trial of one patient proves that a drug works.)

The delays and problems caused by an entirely justifiable suspicion of this stuff don't stop there. Let's say the pilot is a success. Here's what can happen. The regular folks who work at the retailer and don't really have a good feel for statistical concepts and statistical methods now get The Word from the Robed Priests. They reflect on what they're getting and they feel that recent events or their intuitions override what the black box says. Never mind that feelings like this are what make system players go back to casinos. They then go and correct what the black box said.

Then you're in a real quandary. If their corrected version works, then your people are happy with the black box and in fact it doesn't work. And if their correction doesn't work, then they're unhappy.

The long and the short of it is that it's very difficult to tell whether these black boxes work, and if they do work, it's very hard to understand why. Naturally, that's going to delay acceptance. But one has to believe that eventually, if it does work, people will accept it.

Does Science Work?

Does it work? People who have taken statistics courses or had their ears bent by particularly rabid enthusiasts of technical analysis in the stock market know how difficult it is to say. Frankly, I can't tell. For a lot of these things, I think it's very reasonable to believe that you can find significant correlations. But I don't know.

But I think there is another way of viewing it. Even if the algorithms aren't all that good and even if consumer behavior does changed massively and unpredictably from year to year, I think you'd still see some pretty good benefit from Applying Science, because using these applications can help you weed out systematically counterproductive behavior.

Again, take markdown optimization as an example. People hate marking things down. Even experienced people tend to delay the markdowns longer than is optimal, especially on products they like. So, as one software executive told me, any black box that does markdown optimization is likely to produce better results than what people are doing now.

In all the other areas, the same thing is true. Performance by people is often significantly suboptimal, because they are swayed by emotion, can't do everything that is asked, are incorrectly incented, make mistakes, or don't understand what they're doing. If all that Science does is help people avoid silly errors, it will work. It will produce significantly better performance than what is done now.

And if Applying Science works, eventually, the skepticism will be overcome. It may be slowed by mulishness, suspicion, a reluctance to spend money, and a feeling that we can do better than the machine. But it will be overcome.

The Beneficiaries

So who benefits? Well, to some extent, as I have said, the companies that sell high-end Science. Really good software that does replenishment planning, assortment planning, or price optimization is gradually going to find buyers. It will be slow; these days, you don't make a sale without a pilot, and by definition these pilots take a season or more to run. But it will happen.

But I think that over the next few years, the vendors of retail transaction systems, particularly Retek, may benefit even more. There are three reasons for this:

• These vendors also sell Science applications. Retek, for instance, has announced something in each of the areas I've talked about. I have not reviewed these products or services. But, as SAP has shown, even if your product is marginal, your installed base will try to buy it from you if that's at all reasonable. And with Science applications, even a marginal product can produce significant benefit.
• If you're planning on using a lot of Science in a systematic way, you'll need a transaction system that can provide the right data to the Science Applications and can execute the resulting pricing or replenishment orders. Thus, a desire to get a competitive edge by Applying Science will drive at least a few large deals for Retek. (According to one company who asked me not to use their name, this was the major reason that they made a corporate commitment to Retek last year.)
• Science builds on Science. If you discover that Science applications really work for you, that increases your desire for more. If markdown optimization works, getting rapid replenishment of particularly popular items is a natural next step. As people come to realize this, one-stop shopping will become more and more attractive. The largest vendor that will offer this one-stop shopping in the next few years will be Retek.

In the ERP business, the emergence of add-on applications, like SAP's APO and Business Intelligence, had exactly the effect that I'm talking about. SAP made money from selling the add-ons, but they also made money from people who were adding to the core system because they wanted the add-ons to work. And when you bought one, it made buying the rest more attractive.

In ERP, of course, the market for the add-ons was always relatively small relative to the market for the transaction systems. But in retail, the market is relatively speaking quite a bit bigger, because the value is so great. One retailer last Christmas, who had bought ProfitLogic, said that its well above-average results were largely due to the improved markdowns that ProfitLogic suggested. Take that kind of result and add to it similar results from replenishment planning and assortment planning, and the economics of the industry begins to change.

So if Science really does begin to work, the potential market size for companies like Retek will increase substantially.

Clearly, the operative word is "could," not "will." Any number of things could prevent even the moderate acceleration of the Science market that I am predicting, and even if that does occur, any number of things could prevent Retek or the other transaction vendors from benefitting.

It could be that Science never takes hold because retailers can never figure out what to do with the black boxes. The regular folks who do this work may never be able to adapt (though I think it's likely they will), or the retailers will never be willing to hire the talent that can make the best use of the systems.

And it could be that Retek (for instance) will never be able to develop either the applications or the expertise (you need both) that customers need, and the customers will turn elsewhere. You saw this happen at ERP companies who simply failed to come up with adequate SCM, e-procurement, or CRM products even when they saw the advantages of building them.

Some signs that things are panning out the way I predict would include:

• Significant growth for best-in-breed players like ProfitLogic or 4R Systems.
• More deals for transaction systems being driven by a desire to support things like price optimization.
• More stores using broad-band connections to support things like customer loyalty cards and real-time or individual pricing. (It makes more sense to do price optimization when you can control the prices accurately at the store itself.)
• Greater service revenue for the transaction vendors (because installing Science applications is service intensive).
A growth in job openings for statisticians at retail organizations.

Signs that this is not working out the way I predict include the following:

• Stagnation over the next 18 months in sales of retail transaction systems.
• Failure by the transaction companies to come out with plausible Science products during that period.
• Abandonment of Science applications by large companies that have already committed to them.
• A decrease in interest in data cleansing and data synchronization, as evidenced by very slow growth in registrations at, say, UCCnet.

The title of this piece comes from Shakespeare's Henry VI, part III; the phrase is uttered by a character for whom things look pretty gloomy. (He has just lost a major battle). His are feelings that any of us who have tried for many years to get retailers to innovate know pretty well. But you know what? The phrase is uttered midway through Act II. At that point, all hope seems gone. But it turns out, there are still 3 1/2 acts to go.

To see other recent Short Takes, click here for a listing.

Applications with enormous potential. Really.