Feature

In October 1999 12snap's main goal was the development and commercial implementation of the world's first wireless shopping platform for existing mobile devices as well as WAP-enabled phones. The first application allowed mobile phone users across Germany to participate in auctions through a combination of voice, cell broadcast, and short messages, all based on the GSM technology so popular around the world (except in North America).

This auction application went into production in February 2000 in Germany, the United Kingdom, and Italy. Additionally, there were other applications such as fixed-price shopping. In June 2001 a wireless betting application was added for the UK market. At the time of this writing, 12snap has removed the auction application from production until new market opportunities arise. However, we find the application interesting and will discuss it here.

How It Works
The idea is that anybody can participate freely in any of the auctions available. Ideally the participants are people who have already registered with 12snap; non-registered users can still participate in an auction, but if they win they have to register.

The registration process is purposely very easy. The participants can either contact 12snap's call center to verbally provide all their information or they can go to the 12snap.com Web site and type in all of the required information. The important thing is that 12snap ends up with all the payment information to properly close the transaction.

Auctions are announced via Cell Broadcast Services (CBS). The CBS message is received by any users, who activate the feature in their mobile phones. The activation allows configuration and reception of a certain number of channels. Each channel is identified by a three-digit identification and can be labeled. 12snap broadcasts on Channel 123, which is closer to omnicast than multicast.

Every day the application sends several CBS messages to the mobile operator, who then distributes the messages to the cell broadcast-activated handsets. Each message announces the product to be auctioned on that day (see Figure 1), including a basic description of the product, the auction start price, and a phone number pointing to the voice response unit of 12snap (see Figure 2).

As other users participate in the same auction, the system uses the Short Message Service (SMS) to inform previous participants in the auction with "bid topped" messages (see Figure 3). The SMS message carries enough information for the user to participate again and place a higher bid (see Figure 4).

Once the auction is over, the application notifies the winner with a new SMS message (see Figure 5). When the winner is not a registered user, the SMS message also invites the user to register and provide payment information. In this case, the auction application waits for several hours for the unregistered winners to provide their information before proceeding with payment and fulfillment activities. If an unregistered winner does not provide the payment information within that period, he or she is blacklisted and prevented from participating on future auctions.

At last count there were about 80 auctions a week running in Germany. These occurred between noon and 8:00 p.m. on weekdays and between noon and 10:00 p.m. on weekends. The average duration of an auction was about 15 minutes and one hour each day was dedicated to a specific type of product.

Although the auction engine was designed to handle concurrent auctions for the various regions of Germany, this feature was never used.

Administration
The auction application is administered via a browser interface. From the various administrative tasks we can get a look at how an auction is created. This requires the following information:

• Item, supplier number, and name from the dropdown menu refers to the actual product to be auctioned.
• Product Category forces an auction to be scheduled at specific times of the day based on categories, such as "events," where tickets to a music concert could be auctioned, or "electronics" for items such as mobile phones.
• Number of items to be auctioned affects the actual number of winners. If two items are to be auctioned, then two customers will win and will end up paying two different prices according to the strategy followed by 12snap's auction model.
• Starting price
• Warning price is the price above which the customer receives a warning indicating that the amount entered is too large.
• Minimum bid
• Speaker name identifies the set of sound files or voice prompts used by the IVR to speak out the welcome messages, instructions, and product descriptions.
• Item type affects the timing parameters of the auction, particularly those affecting the closing of the auction and processing of payment data for winning customers. For example, an item could be perishable.
• Payment methods After this, other screens are used to add more information related to a new auction:
• The date and the starting time
• The time the auction will last and extension time
• Assign a time scheduler to the auction.
• Assign a slot to the auction begin time.

The Whole Picture
Putting it all together, we have all of the components shown in Figure 6, where the participants supporting the process described earlier are:

• The customer using a handset or a browser
• The public land mobile network (PLMN) of the mobile operator
• The mobile operator
• Public switched telephone network (PSTN) for voice hand-over to the TSP
• Packet data network for connection to the SMS center
• The telephone service provider for SMS and IVR capability
• 12snap systems
• The payment partner
• Logistics for delivering goods to the winners
• Address-checking facility for maintaining high quality on data from registrations

The Solution
The 12snap team had a major challenge in trying to design, develop, test, and implement the application in about four months, which is more or less equivalent to the infamous "yesterday" deadline that we are all too familiar with. Additionally, the application needed a cost-effective integration solution capable of supporting the connection needs to systems, such as voice response units, SMS gateways, and third-party payment services.

Java, along with the Enterprise Java Beans specification, was quickly identified as the basic technology that would allow a quick development and implement a clean service-based solution. WebLogic was then selected as the application server platform mainly because it worked correctly on the first try.

The design stage was divided into the following parts:

• Auction logic
• Software architecture
• Data modeling
• Front-end design
• Call-flow and IVR integration
• SMS and CBS integration
• Development setup
• Production setup

Timed Auction Logic
The initial aspects of the design addressed the auction logic itself. There are a variety of auction models that can be classified based on three phases: setup, bidding, and resolution. For example, in Dutch auctions the setup phase initiates a very high opening bid. During that phase the participants bid to lower the initial price. This will run until demand rises to match supply or the goods are exhausted.

12snap adopted a very simple auction model. In the setup phase the number of goods is defined. If there is more than one item, the winners will pay different prices. During the bidding phase the participants bid to increase the price. This process stops when no participant places a bid during a period called the extension time, which is defined in the setup phase. When there is only one item, the resolution phase assigns the item to the highest bidder. When there is more then one item (N > 1), they are assigned to the N topmost bids.

In addition to the basic strategy, a number of states were defined for the auction, which indicate what methods need to be triggered. The actual transitions are timer-based and use WebLogic timer classes. Although these are WebLogic proprietary classes, there is no equivalent functionality present in the J2EE specification.

The problem is that these timer classes are not cluster-aware, that is, a WebLogic timer is not aware of other timers running outside of its JVM. Since the application would run in a WebLogic cluster, this was going to be a show-stopper. The 12snap team had to perform some magic to allow the timer to work in a WL cluster configuration. This magic was based on three EJBs:

• The AuctionTimer: An entity bean that contains the scheduling for an auction. The scheduling is a vector of events.
• The AuctionTimerMgr: A session bean responsible for creating, cancelling and rescheduling the AuctionTimer
• The TimerSchedulerMgr: A session bean responsible for accessing the WebLogic TimerService

And there were also some normal Java classes:

• The AuctionTimerScheduler: The kernel of the system. It implements some specific WebLogic Timer interfaces and is the one called by WebLogic for scheduling and triggering events. Each instance is associated with an AuctionTimer instance. For scheduling, it calls the AuctionTimer to get the next activation time (but not for the first one, to avoid transaction locking). For triggering, it gets the event from the AuctionTimer, and removes it from the list of next events. Then it gets the action for this event, which is the name of the method to call on the AuctionMgr.
• The TimerSchedulerHelper: Provides utility methods for registering and cancelling AuctionTimer. It also provides methods for sending multicast messages for scheduling and cancelling timers.
• The TimeServiceHelper: Provides some utility methods to access the WebLogic Time Service.
• The TimerSchedulerMulticastThread: A thread whose role is to listen for multicast messages and execute the associated action.
• The SchedulingValue class: Describes an event.

There are other classes used, but they are not worth discussing. You can download this package from www.wldj.com.

Functional Overview
In a cluster, all the members will have a local auction timer. Every event for every auction will be scheduled on all the machines. When a timer wants to execute an event, it first checks to see if the action has already been executed. If so, it does nothing; otherwise, it executes the action.

Execution of an Event
When an event is triggered, the database row concerning the particular auction is locked by using a "select for update." This is done on the t_auction_event table.

It then gets the next event to be executed, first checking that the event has not already been executed. To do so, it compares the date of this event with the next activation date stored in the AuctionTimerScheduler for when this event was scheduled (previous call to the schedule method). If these two dates match, then it can be executed. If not, another machine must already have executed this action, so the method just returns.

By locking the row in the database, it prevents two instances of WebLogic from executing a sequence at exactly the same time.

The transactions are handled by using a "user demarcated" transaction using JTS.

Scheduling on all the machines
This is the trickiest part of the system. As there was no way to know all of the instances of WebLogic servers that make up a cluster, 12snap was forced to implement a solution to communicate with all the members of the cluster.

This was done by using multicast and a dedicated socket. There is a thread that listens on this socket for messages and executes the commands. On the other side, when you want to schedule or cancel a timer you simply send a message on this multicast socket and every member of the cluster will receive it.

The TimerSchedulerMulticastThread listens for messages. The TimerSchedulerHelper contains utility methods to send these messages. The TimerSchedulerMgr bean, which is responsible for scheduling or cancelling timers, sends only the multicast messages.

Message format
The message is a byte buffer with the format shown in Table 1.

Properties involved
The system used the following properties:

• timerScheduler.multicastIp: Multicast IP for the timer system
• timerScheduler.multicastPort: Multicast port for the timer system
• timerScheduler.sessionTimeout: Transaction timeout for the execution of an event

Restarting the timers
This process is different from the basic scheduling because when it is executed the multicast socket may not be open. Basically, every machine registers the StartTimers startup class. But now, when you update and register the timer, you lock the row in the database with a "select for update" and a user-demarcated transaction. Then you update, that is, you remove the events that could not be executed, and you register the timer locally. To be able to schedule a timer locally or cluster-wide, some methods have been added to AuctionTimerMgr.

State Transitions
Equipped with this sophisticated timer, the application can control state transitions for various auctions running concurrently.

In terms of modeling the auction engine, failures should receive special attention; they did not consider, for example, an "error" state for failures related to payment.

Software Architecture
Implementing the core auction logic with WebLogic and EJB 1.0 led 12snap to make a few key decisions. It chose a now very common pattern that allowed it to deal with services for later reuse. Services would be implemented using stateless session beans. For persistence, they chose entity beans using container-managed persistence - at least for most tables. Servlets would be used as access method for its services. This is illustrated in Figure 7.

Presentation Layer
When the project was started, JSP had barely made it to 1.0. Considering the difficulties in keeping up with the specification and the known pitfalls in maintaining production JSPs, 12snap chose a more conservative template approach to the presentation layer and decided to use FreeMarker, which comes in the form of a library with template classes.

Short Message Services
The Short Message Service (SMS), as defined within the GSM digital mobile phone standard, has several unique features. A single short message can be up to 160 characters of text in length and consists of words, numbers, or an alphanumeric combination. Non-text- based short messages (for example, those in binary format) are also supported.

The Short Message Service is a store and forward service. In other words, short messages are not sent directly from sender to recipient but through an SMS Center (SMSC) instead. Each mobile telephone network that supports SMS has one or more messaging centers to handle and manage the short messages.

The Short Message Service features confirmation of message delivery. Unlike paging, users do not simply send a short message and trust that it gets delivered. Instead, the sender can receive a return message notifying them whether the short message has been delivered or not.

Short messages can be sent and received simultaneously with GSM voice, data, and fax calls because whereas the other calls take over a dedicated radio channel for the duration of that call, short messages travel over and above the radio channel using the signaling path. As such, users of SMS rarely, if ever, get a busy or engaged signal.

Sending and receiving SMS messages from an application requires a gateway solution that implements the native protocol supported by the SMS Center. On the application side, SMS gateways tend to offer standard access protocols such as SMTP, RMI, HTTP, or file access to interact with the mobile message infrastructure. Other formats are also available.

Initially, 12snap decided to outsource the SMS gateway technology to one of the mobile operators in Germany and agreed on an SMTP interface. At the WebLogic level, they used JavaMail to prepare and send e-mail messages to the SMS gateway that corresponded to the SMS notifications the auction engine needed to send to the auction participants.

Cell Broadcast Services
12snap initially implemented a gateway to the Cell Broadcast Center (CBC) of one of the main operators in Germany by using its Cell Broadcast Entity (CBE). It was a solution based on a Web server hosting forms processed in the back-end by an application server that would, in turn, pass requests to the CBC. That n-tier solution was hosted at the operator premises. Eventually, the solution was migrated to 12snap's own WebLogic servers by implementing the ASN.1 protocol required to directly access the CBC (in agreement with the operator as there are some very sensitive load and regionality issues to consider).

The implementation of the ASN.1 protocol was made using JavaCC and a reference CBE tool from the CBC manufacturer.

Interactive Voice Response
Interactive Voice Response servers (IVRs) are curious animals when you have an application server view of the world. After a 20-minute discussion with IVR specialists, 12snap discovered that the application server and the IVR can easily compete for the business logic decision in processes. From the very beginning they tried to confine the business logic decisions to the WebLogic cluster and keep the IVR as "lean" as possible. They were in fact lining up for the upcoming VXML, VoxML, and VoiceXML standards without really knowing it.

An IVR has connections on one side to the telephony network (e.g. ISDN Q.931) and on the other side to the data network. Its DSP boards and software provide the necessary glue between both. When a call arrives at the IVR a script is immediately executed and two key pieces of information are given to that script: ANI and DNIS. ANI stands for Automatic Number Identification and represents the number of the person calling the IVR. The Dialed Number Identification Service (DNIS) carries the number called by the person. In essence, these numbers are equivalent to routing information that can be used by any script to make decisions.

The scripts running on the IVR are commonly referred to as call-flows and link voice prompts together. It was decided, for performance reasons, to keep the actual sound files on the IVR itself. Call-flow decision points ask for a link to the middle layer for consistency across multiple channels. For example, when a caller dials into a specific auction, the IVR will need to contact the WebLogic server to find out if that auction is available or not.

12snap auctions began by hosting its equipment at an ISP that offered managed IVR solutions. In order to avoid technology dependencies it integrated the IVR with WebLogic through HTTP by implementing a basic thread-safe browser library on the IVR. For each line the IVR picks up, a browser thread was activated and used by the call-flow scripts to place HTTP requests against WebLogic. Servlets such as GetBid were implemented on the WebLogic cluster to pass information back to the call-flows.

The External Server
Initially, the application requirements defined the support of various payment methods (credit card, direct debit, and cash-on-delivery) as well as address verification and fulfilment notifications to be sent to its logistics partner. Integrating with the business process of such service providers depends on the technology available. The payment- and address-checking services were initially available from the same provider in the form of shared objects for SPARC. The fulfilment partner used a much simpler interface: e-mail. It decided to implement, for those third-party libraries, a special component dubbed "external server," which would run in a separate JVM and host the JNI-wrapped libraries.

This component supports modules, in this case Java objects that access native libraries. Accessing these libraries from the WebLogic JVM was considered too risky due to JNI-related issues, such as instability.

The external server registers a Factory with WebLogic's JNDI tree so that they can access the modules from the application server. A "check" thread acts like a watchdog and verifies that the connection to the Weblogic server is still alive. If not, it re-registers itself into the JNDI tree. A console allows operators to administer the server via telnet.

Development Environment
In order to identify the most suitable platform for its system, 12snap compared NT and Solaris systems. Based on the anticipated transaction estimations (accumulated values across IVR, Web, and WAP), it opted for Sun Solaris as the development and production platform. Since this is expensive equipment it had to be a little creative to minimize costs and agreed on the configuration shown in Table 2.

Linux Workstations Setup Procedure
Once configured with basic software and networking, they downloaded the customization script for the workstation from the repository server using FTP and ran the install shell script located under the install directory. This script performs the following actions:

1. Creates special directories
2. Backs up configuration files (profile, hosts, fstab)
3. Installs skeletons
4. Installs Java Development Environment (JDE) for Emacs
5. Mounts file systems
6. Installs JDK

The tools related to the installation script are:

• Libraries (e.g. FreeMarker)
• Application server (e.g. weblogicaux.jar and Service Packs)
• JDK

Additional tools are shown in Table 3.

When it started the service, it identified a mobile operator capable of supporting it with hosting, IVR, SMS gateway, and call center capability. It was left to bring in its production hardware as shown in Table 4.

Figure 8 shows how the components are laid out on the network segments.

JDK 1.3 and WebLogic 5.1 Migration
A few months ago 12snap decided to revisit the WebLogic cluster configuration and set up some tests under new conditions - namely upgrading the JVM from 1.2.2 to 1.3, upgrading to WebLogic 5.1, and checking out the performance of two instances of WebLogic per server instead of one. These tests were conducted by benchmarking the performance of four of the most used servlets in its application:

• GetAuctions: Used on the Web site to display a list of auctions
• GetCategories: Used by the IVR to present a menu of auctions running
• GetCurrentPrice: Used by the IVR to tell the latest price for one auction
• SetBid: Used by the IVR to set a customer's bid

These benchmarks showed an increase in performance coming not only from the migration to the latest JVM but also from having two instances of the WebLogic server on every Sun machine in production. This effectively created a cluster of four WebLogic instances on two Sun machines. The benchmark was done by simulating 120 simultaneous users generated by The Grinder. This is an open source Java-based load generation tool. It can be downloaded for free from http://grinder.sourceforge.net.

Day-to-Day
The following is a high-level list of some processes that take place on a day-to-day basis to run the wireless auctions:

1. Product scouting (at the beginning of the month)
2. Order fax sent to supplier
3. Production and uploading

• Get product pictures for Web site.
• Get short and long description of products.
• Produce the IVR voice prompts of the long and short description.
• Upload the files.

4. Finalize production
5. Auction planning
6. Auction scheduling
7. Last-minute changes (e.g. new products)

Common Scheduling Problems
The most common error in scheduling auctions was the "wrong category for this hour." Someone already scheduled an auction for a specific hour and the new auction is in a different category from the first one of that hour.

The second most common error was scheduling auctions which should start as soon as possible. The auctions were coupled to the Cell Broadcast channel and specifically to the property CBCDelay and the value "Preceding Time for Hourly Reminder." A large value here allows the broadcast to be sent early (which is better from the CB Center perspective), but this also means that once this time had passed it was too late to schedule a new auction for that hour.

After scheduling an auction, all CB messages were sent to the CB Center immediately. In the case of an error a delete-message was sent. At this moment, CB is available only in Germany. It implemented an SMS-based subscription and notification module for use in Italy and UK.

Operational Information
Before closing the auction application, 12snap had about 80,000 subscribers in Germany, 50,000 in Italy, and approximately 10,000 in the UK. These numbers represent the number of customers for which 12snap has full information to complete payment transactions.

As we mentioned earlier, the system did not require the users to be registered to participate in an auction, so the potential users were actually higher than the number of registered users.

All servers for all countries are hosted at an ISP in the UK. Each country has its own combination of WebLogic proxy and cluster servers. Database servers are shared for two of the three countries. The ISP provides connectivity to the 16 ISDN E-1 lines to service a total of 480 lines.

There were about 80 auctions per week starting at noon and, interestingly enough, the peak times were at noon and at around 6 p.m.

Lessons Learned
In the day-to-day operations, one of the first things 12snap learned was that whenever there was a failure, it had to look in the WebLogic log file. A careful analysis of this file solved 90% of the problems it encountered. This is a hint that is placed conspicuously in all of the "Hint" documents.

One of the first comments from its software engineers was, "We need code-beautifying, because code is partly not readable." This was an annoyance that was never resolved. It would be helpful to have some sort of beautifying tool for Java and specifically for J2EE.

Within its code, exception handling suffered from poor design. 12snap limited its analysis to splitting exceptions into Recoverable and Unrecoverable from the user standpoint, e.g., wrong input: recoverable, database down: unrecoverable. This falls short when it comes to choosing which type is needed by the developer.

12snap now realizes it should have adopted nested exceptions based on a Runtime/Checked exceptions categorization. The output of exceptions should only be done in servlets or timer-triggered methods.

The proper naming of variables was another area that caused trouble and 12snap concluded that it should always be carefully considered during code reviews. For example, at some point 12snap had the following "pathological" variable names:

• available meaning pre-bidding
• maxPrice meaning stopPrice
• ready meaning scheduled
• closed meaning ended

For the ERP system 12snap chose Navision Financials running on NT. In order to integrate the WebLogic cluster of a specific country and the Navision server, it installed and configured a transfer server, (which was a dedicated Sun server), with Samba. This server was installed and configured to belong to the same domain as the ERP server. As soon as the session beans completed successful payment and fulfilment transactions, a file was written into that special directory using the WebLogic File Services. This was too simple.

Writing the file could fail for several good reasons (e.g., disk full), and information would not pass to the ERP system although the application would flag it as complete. In retrospect, this is something that 12snap reckons it should have implemented using an asynchronous mechanism, such as the Java Message Service. This way there can be some level of guarantee that the transactions will arrive correctly in the ERP system.

From Navision's side, a dedicated client runs the Navision data ports, which are timer-controlled scripts that would fetch or deposit files to/from Navision.

Conclusion
Although the application does not use all of the J2EE APIs, mainly because of its availability at the time it was developed, 12snap was able to develop and put it into production in record time. It ran for more than 18 months on version 4.5.1 with a relatively large number of users taking full advantage of WebLogic clustering. One of the neat tricks was getting the timers to work in a clustered environment-and we have shared that experience here.

As you can see, nothing magical happened in order to have a stable application running. Just a lot of common sense and a few mistakes that we hope everyone can learn from.

Add Your Feedback

In order to post a comment you need to be registered and logged in.

Register | Log in

Please wait while we process your request...