Performance Roadmap

Isolating the cause of why a machine is slow can be tiresome and hard. One machine might function perfectly, while another performs poorly. Sometimes the Point of Sale applications will show slow performance but arent actually the software causing the problem. This document covers a number of test to try and areas to investigate.

This sheet only covers slow performance, it is not designed to help isolate non-functioning systems

Quick Tips

1. Run Task Manager, select processs tab and check how much CPU is being used while "idle" and what processes are using it. If your CPU is not idle when it should be, this can cause problems. Common examples are iTunes, which can consume all CPU resource in the background.

2. If you are on a network system (ie remote database), check the general network health. Open a CMD prompt, and type
   ping -n 20 -l 2000 host
   You should not see any packet loss, and time per packet should be reasonably constant, ideally <1ms, although times up to 20ms can be OK on WANs. Times greater than 100ms will probably start to cause performance issues. more

3. If your problem is Point Of Sale related, try using the QuickCode f600, which is the command to clean old files that are no longer required. You do not need to run this more than once every couple of months, but will have no ill effect if run more often.

4. Verify the disk has sufficient free disk space, AND empty the recycle bin

5. If you are using Fieldpine Point Of Sale, you might like to check your TEMP folders (these are Windows folders, not Fieldpine related) for large numbers of files. See kb2105.htm for more details.

6. Check that Anti virus scanners aren't consuming all disk IO on the system. Often they can drain all available IO bandwidth without using much CPU.

Point Of Sale Specific

Area	Possible Causes
Multiple	Verify indexes exist on major tables. Sales.sid (primary key) Sales.completeddt (non unique) Salelines.sid, Salelines.seqnce (primary key) payments.sid, payments.sequence (primary key) products.pid (primary key) eftposreceipts.eftrid (primary key. Caution however as primary key on StoreServers or HeadOffice servers can differ) productvariant.pvariant,basepid (primary key, IF table in use, and this is not typically enabled) You can use Quickcode 71242 to check indexes
Multiple	Verify secondary indexes exist. These indexes aren't as critical but can improve performance, especially where the table mentioned is large in size. products.plucode (if selling by plu is frequent) barcodes.barcode (for sites with more than around 30,000 barcodes defined) customers.cid (primary key) eftposreceipts.sid (if integrated or interfaced EFTpos is in use) products_inventory.(pid,location_id) (for sites with more than around 10,000 product lines. This index typically exists as part of the primary key) auditsystem.asid (primary key) cashdrawerlog.opendatetime (if table has lots of entries) custcap_store_log.regno (if table has lots of entries AND registration numbers are collected)
Using Past Sales is slow	Verify indexs exist on Sales.startdt (non unique) and sales.completeddt (non unique) or that setting PastSalesDateOrder is set not to use Start Date for selection. The default operation at most sites is to scan by both date ranges.
End of Day slow	Ensure that eftpos.receipts has an index on sid. Ensure that cashdrawerlog.opendatetime has an index Make sure the settlement period is not too large (few days maximum)
Reward/Loyalty programs	Ensure that rewardtxns.(rpid,cid) index exists (primary key)
Prepay Cards	If heavily used, verify primary index svccards.barcode If heavily used, index subscriptions.pid
Subscriptions and Stock Holds	If heavily used, index stockholdrequests.pid If heavily used, index subscriptions.pid
Selling Tickets is slow	Ticket sales made via the main ticketing screen (one with buttons for each session) are processed remotely. Performance is therefore around local and remote machines. If the screen is slow to appear Network performance to ticketing server. The ticketing service may be overloaded or suffering from blocks on the database. This process is extremely time critical. If the screen is slow to use after it has drawn Look for local CPU problems Your pricing structure may be too complex. If you have a large number of special combo prices the system needs to search every combination to determine the best price for the consumer which uses quite a bit of CPU resource. (This is known as an NP problem in computing)
System is just sluggish	Does the problem affect all operations on the box? Do you have the drives "shared" in Windows, and are users actively using it? (Advanced) Check that your controllers are not running in "PIO Mode" rather than "DMA Mode". Search the internet for specific instructions for your Windows version, but most systems are checked from the IDE Controllers in Device Manager - there is a tab called "Advanced Settings" (Advanced) Verify that background jobs arent using all network sockets. Issue the command "netstat -na" from the command line and check that you get a typical number of lines/connections. Most sites only run 1 or 2 pages of links, although this depends on what the system is doing. Verify that disk IO rate and bandwidth is not at capacity. Common causes for this are background antivirus scans. Windows 7. (Advanced). Try temporarily stopping the "Background Intelligent Transfer Service". On some systems stopping this service greatly improved system performance, even though there was no outward sign of problem from this program. If this helps your system we suggest you contact your IT support for long term advice. Is your hard drive starting to fail? Symptoms of this can include long periods (30 seconds) of no useable disk IO while the drive attempts to recover bad sectors. You can use chkdsk with the bad block scan switch to check this. (Advanced, Rare) Are the level 1 and Level 2 (L1 & L2) caches disabled in the BIOS? Without these caches systems perform slowly for all use. Is the system running a web browser, and has it been open for a couple of hours? Try restarting the browser application and see if performance increases inside the POS. This is possible if the system gets slower as the day progresses. Try running a hardware level performance test using QeHardware this might show undetected issues with hardware affecting performance. This is especially important on VM environments.
Shared database issues	(Advanced) Verify network file access performance is good. From a slave lane try and edit a file in the same shared folder as the database. This should be fast to appear. A symptom of this is very slow startup time of the POS Access. Database compression required? SQL/Server. indexes may need rebuilding to remove fragmentation Oracle. A commpress/recover operation may be required. ODBC. Has logging been enabled on the ODBC DSN? This slows overall operation greatly. Is the system slow on just slaves or masters also? If just slaves, then check indexs and network throughput. Acccess. Is the database opened by another user on the network, by POS or Access, and they are not on the local LAN? This can cause a lot of network traffic over slower network links than local LANs operate. SQL/Server. If you are experiencing hangs of a minute or more, check to ensure that: "Snapshot isolation" and "Read committed snapshot" are enabled for the database inside SQL/Server. These settings alter how record locking and transactions co-exist. (Right click on database name, properties, options). Contact your DBA for advice in this area. Verify that "dbcdao:isolate=1" is set inside fpos.ctl

Global Data Server Specific

In general, GDS performance is driven by the underlying database engine, as the vast majority of queries are sent to the database to process.

Area	Possible Causes
Management Connections Screen slow to display	This is mainly caused by DNS servers. See Kb4 Article for further information.

Network

If your POS environment requires a network for operation, such as shared Access databases on another machine or ODBC connections to a remote database server, then the network performance between that machine and POS nodes is important.

If two machines are on the same LAN, issue this command

ping -n 20 -l 50000 ....

On a 100Mb/sec LAN you should see response times of 8mS if the network is functioning at full speed. There will be some variation due to network loading at the exact instant ping sends the data on the wire.

If you are using shared databases and having problems on a slave computer, try creating a file in the shared folder where the database resides. If the database resides on Z:\Pos\Green.mdb, then from a command line, try this command

notepad "z:\Pos\temp.txt"

The notepad editor should appear almost instantly. Enter some data and save the file. It should save instantly also. (Don't forget to delete temp.txt). If the edit is not instant, then you have a networking or Windows share issue outside the POS and should contact your IT support team. Some known causes for this issue have included faulty network cables, faulty switch and incorrect DNS setup

Hardware Environment

The following is the technical hardware performance needs (advanced)

• Gds/2 prefers machines with high main memory bandwidth. Some functions are optimised towards large sequential scans of main memory.
• IO performance, both disk and network, are critical to perceived user performance.
• Disk IO workloads from Gds are a mix of random IO (typically small) and large scale sequential reads. Sequential reads use overlapped IO and may cause IO queues to form on drives for brief periods.

Virtual Environments

More: Virtual Environment Operation

If you are using Gds/2 in a virtual environment, do not over provision resources. Gds/2 scans the available hardware and will make decisions about processing mode based on this information. If you have 2 physical cores but create a VM with 4 cores, Gds/2 may at times select to run 4 cpu intensive loads in parallel. Equally, do not cause instructions to be emulated. Gds/2 uses CPU dispatching to select the optimised techniques for different hardware.

Running highly loaded Gds/2 in different VM on the same physical machine may generate high traffic on the cache coherency control bus between sockets in the CPU. By highly loaded, we mean sustained loads of hundreds/thousands of requests/sec. If you have this need, consider isolating the VMs to different sockets to eliminate the traffic

Hyper-V Virtual Environment

• Ensure that VMQ is configured correctly, especially for Gds/2 or if ODBC database connections are being used. Network performance is critical to overall performance
• Enable Jumbo frames if at all possible

Xen Server Virtual Environment

• IO performance in Xen is critical. Ensure IO paths are optimised. Use the "Resource Monitor" to verify that disk latency is low. Average IO latency should be below 20mS as a rule of thumb.

VMware Virtual Environment

• Ensure you are using a release of VMware released after 2014. If you are using a version before this you may experience rare issues with mapped sections.
• VMmove has not been fully tested by Fieldpine. If you do use VMmove, you must ensure that hardware ids, such as network MAC addresses, do not change dynamically.

Terminal Server

Running Gds/2 as a service in a Terminal Server environment is not recommended for maximum performance. It is supported but performance may be affected. This a technical distinction and most users will not observe issues. The performance discussed below is about fractions of a second per query. It reflects more on the absolute extreme performance, rather than general overall performance. You will not achieve the maximum throughput and more than one actively processing user can negatively affect others.

A Terminal Server environment runs multiple desktops inside the same Windows Environment. High load in one desktop can cause performance slows in other environments.

If you are using both the Gds/2 Excel Addin and Gds/2 inside the same terminal server environment, you can expect performance problems and lowered maximum throughput. Both Excel and Gds/2 start resources to maximise the use of available cores, which leads to over commitment of resources.

Gds assumes it is operating on a dedicated server and takes steps to optimise this for maximum user response. This optimisation is not optimal if user sessions are running in the same environment. If Gds/2 detects Terminal server it will automatically switch to low priority mode. In low priority mode, all other CPU load, from any user, will take priority over Gds

If you are running Terminal Server in a virtual environment you may wish to configure the VM to have a larger number of cores, but limited cpu use. ie Setup the VM as an 8 core low spec chipset, rather than a 2 core high spec chipset, (but do not over provision)

THIS ARTICLE APPLIES TO

• All Fieldpine Products. As at November 2011