Test Plan for Robot Testing of Spine Effect

In this draft document, any point at all is open to discussion and modification. But the points in red must be resolved; our goal must be to have no red issues before the clubs leave for the robot test facility.
Dave Tutelman

1. Team Membership

Alan Brooks	Vibration test engineer, LLNL
Bob Dodds	Technical Director, PCS
Don Johnson	Independent golf clubmaker
John Kaufman	President, Kaufman Enterprises ("Club Scout")
Dave Tutelman	Independent research engineer

2. Objective

The goal of this testing is to determine some quantitative aspects of the spine effect, notably:

• Performance effects of shaft orientation such as distance, direction, and statistical spread of distance and direction.
• The degree that these effects depend on the magnitude of the spine.

It is also worth noting some plausible would-be goals, that are not objectives of this set of tests:

• Effect of spine on "feel" (whatever that is). This cannot be determined by robot testing.
• The best method of determining the spine. We will choose shafts for which FLO, spine finders, and PUREing give essentially the same positions on the shaft.
  
• The best orientation for the spine. This could be done by robot testing, but we probably do not have funding for enough testing to do it. We will use the "best current practice" for our testing, and assume it is correct. Note that there are two credible candidates for "best current practice"; we may or may not be able to run enough tests to compare these two.
• The so-called "supershaft" effect: that is, a properly-aligned shaft with a very large spine compared to a properly-aligned shaft with little or no spine. This could be done with robot testing, but would probably need more tests than our budget will support.
  
• Effect of uniformity of alignment from club to club within a set. This is probably very hard to do with robot testing, due to:
• The possibility that such differences might be masked by unavoidable other club-to-club differences.
• Our funding supports the testing of only a few clubs. We are choosing otherwise-identical drivers with differing spines, as probably the club most sensitive to spine alignment.
• Effect of shaft bend. We will select shafts for straightness and "Type 2" behavior, in order to isolate spine effects from bend effects.
  

3. General Approach

We have been funded by Harrison Sports Golf Shafts ( http://www.harrison.com/) for a single robot session for this work. The robot lab is Golf Laboratories ( http://www.golflabs.com) in San Diego.

We will conduct tests on clubs with:

1. Different alignment of the spine.
• Spine in the best position.
• Spine in the worst position.
  
2. Different magnitude of the spine.
• A large, prominent spine.
• A modest-sized spine.
  

Except for these two factors, the clubs will be as identical as we can make them. The clubs will be drivers; the motivation is that the driver will probably show the most sensitivity to component "deviations" such as spine alignment. The tests will be for otherwise identical conditions of face angle, center hit, clubhead velocity, ball characteristics, and any other test conditions that we can control.

We will measure the landing point (distance and direction) of each ball hit. With this data in hand, we can process it to determine central tendencies and statistical variations for each club.

As per the "Objectives" section, this should allow us to quantify the effect of a spine on performance, and the effect the amount of spine has on performance differences due to alignment.

4. Constraints and Assumptions

1. We have funding for one robot session.
2. Golf Labs feels that you need at least 20 hits for statistically significant results. (Alan Brooks has an action item to review this with Golf Labs; his initial thoughts on the matter are in the Notes. We could do one more test if 16 or 17 hits is sufficient. We temporarily have two plans, one if we can do the extra test and the other if we cannot.)
3. Due to weather conditions, we probably have time for about 90 hits. Therefore, we have an opportunity to compare four different test setups of 20 hits each, or five different test setups of 17 hits each. (This constraint is imposed by the usual wind patterns. We will start early in the morning, when the wind is calm. The data will become much less meaningful if/when the wind comes up. So far, we have made no attempt to plan the tests so that we can either get more reliable data or answer other questions if the wind holds off longer than expected.)
   
4. It is hard to characterize shafts such that we are sure everything about two shafts is the same. (This will lead to some test clubs having the same shaft in different alignments.)
5. It is almost as hard to characterize clubheads such that we are sure everything about two clubheads is the same.
6. There are different ways currently in use of determining the spine of a shaft. (See the next section.)
7. There are different theories about the best alignment of the spine. (See the next section.)
8. There is fairly general agreement that the worst alignment of the spine is 45 degrees to the target plane at impact.
9. There is currently no understanding nor theory to determine what shaft flex nor tip stiffness is the most sensitive to spine effect. However, there is some indication that a shaft that is torsionally flexible or tip flexible will increase the effect of the spine on performance.
10. We will aim for statistically significant results to a precision of 3 yards of distance and 3 yards of direction, with a confidence level of 85%. (These numbers are placeholders; they have not passed any consensus. Moreover, since they affect the number of tests we can do, we may change them if the implications cripple the test plan.)
    

5. Finding and Aligning the Spine

As noted above, there are several different approaches recommended for finding the "significant" plane of the shaft, and also different approaches for aligning it once it is found. Here is a discussion of these differences, and how we will deal with them in order to gain maximum information from the limited testing we can do.

5.1. Approaches to Finding the Spine

Techniques for identifying the stiffest and softest planes of a shaft include:

• Deflection boards and variants, to determine the stiffest and the softest directions of the shaft. They can be used for:
• Raw deflection measurement with a fixed weight, looking for the directions of maximum and minimum deflection.
• Incrementally adding a load to the shaft and looking for the directions of maximum and minimum incremental difference in deflection due to the load. This is more accurate, as it cannot be fooled by a bent shaft.
• Frequency meters, to determine the highest and lowest frequency directions of the shaft, which correspond to the stiffest and softest directions, respectively. This method can also determine Flat Line Oscillation (referred to as FLO), which should occur in the high- and low-frequency planes.
• Bearing-based "spine finders" that depend on the feel of the shaft as it is rotated. The theory is that the shaft will want to rotate out of being bent in the stiffest plane, and will want to rotate into the softest plane. The theory works as long as the shaft is straight, the outer surface of the shaft is round in the vicinity of the bearings, and the stiff side stays in more or less the same direction throughout the length of the shaft. If any of these criteria is not met, then the frequency method or differential deflection method is more reliable.
  

5.2. Nomenclature

We must deal with (and try to reconcile) several different systems of nomenclature. An important community of custom clubmakers (exemplified, for instance, by the Spinetalk forum) have their own terminology, described in a document by Bill Day. There are also two commercial spine location services, and each has its own terminology.

• The Spinetalk nomenclature refers to the stiff direction of the shaft as a "spine", or S for short. The soft direction (the direction the shaft wants to assume in a spine finder) is referred to as the "neutral bending position", abbreviated NBP or N. This school describes shafts as Type 1 or Type 2, based on the number and alignment of S and N:
• A Type 2 shaft has two S directions and two N directions. The S directions are 180° apart, as are the N directions. In a well-behaved Type 2 shaft, the S and N planes are 90° apart.
• A Type 1 shaft has a single S direction, and an N direction about 180° away from it. At least three members of our team are convinced that a Type 1 shaft is a bent shaft, and that a proper differential deflection measurement will show it to be a Type 2 in actuality.
• The commercial spine-alignment company SST offers to "PURE"® the shaft. Clubmakers who have measured shafts that have been PUREd feel that the recommended direction is the FLO plane associated with the stiffer direction.
• The commercial spine-alignment company ASD offers to "PEAK"® the shaft. Fairly extensive measurement by clubmakers of shafts that have been PEAKed show that the recommended direction is simply the NBP to a high statistical significance.

Engineering considerations strongly suggest that all "well behaved" shafts are Type 2 shafts, and that the S and N planes correspond to FLO planes. Where this is not the case, one or more of the following is likely:

• The spine/NBP was found with a feel-based spine finder or simple (not incremental) deflection measurement.
  
• The shaft is bent.
• The stiff/soft orientation of the shaft changes position substantially along the length of the shaft.
  

5.3. Alignment

Having discussed the different systems of finding and naming the directions of a shaft, we still have not discussed how to align the shaft with respect to those directions. Again, there are a few schools of thought:

• Some say the S plane should be down the target line at impact. It would appear that this is SST's strategy, though they do not state it explicitly. In the rest of this document, we will refer to this as "S-plane alignment".
• Some say the NBP should be down the target line at impact. This is the most popular position among independent clubfitters, though some prefer the S-plane approach. It is also clearly ASD's strategy, though they do not state it explicitly. In the rest of this document, we will refer to this as "N-plane alignment".
• All agree that the worst possible alignment is with S and N planes at 45° to the target line at impact.
  

6. Test Facility

To be supplied by Alan Brooks after visit. Tentative date of visit is April 22, 2003.

7. Test Plan

7.1. Preparation

Because of the difficulty of matching everything about a club except the spine, we will build as few clubs as possible with as few distinct components as possible, re-using the same components -- realigned -- in multiple clubs. There must be at least two clubs, because an important objective is to determine the effect of magnitude of the spine. Therefore, we need at least two different shafts with different magnitudes of spine. Call these "shaft A" with a large spine, and "shaft B" with a modest spine. The plan is:

• Build a club from head A and shaft A with the best spine alignment (call this club A1).
• After testing club A1, reorient shaft A in head A to give the worst possible spine alignment (call this club A2).
  
• Build a club from head B and shaft B with the worst spine alignment (call this club B2).
• After testing club B2, reorient shaft B in head B to give the best possible spine alignment (call this club B1).

Considerations for the order of these tests is discussed in the Notes.

Thus for two different shafts (two different spine magnitudes) we are able to compare the same head and the same shaft in two different alignments. We accomplish this by comparing club A1 with club A2, and club B1 with club B2.

Note that it is unlikely that we can compare club A1 with club B1, or club A2 with club B2. If no valid comparison can be made, we will not have data to test the speculation (sometimes called "supershaft") that a spine-aligned club with a large spine gives better performance than a spine-aligned club with a small spine or no measurable spine. The most important reason that we are unlikely to get comparable data is that the shafts have differences in the highest and lowest frequencies. It is hard to talk about them being "frequency matched" if you don't know which frequency (highest? lowest? average of the two?) is significant in doing the matching.

We will build two different drivers in advance, to the following specifications:

1. Identical heads. 
• Mike has told me that most OEM heads have a more consistent placement of the center of gravity than most component heads. Bob maintains that the cast Ruger heads are more uniform in performance than any forged head. Mike has indicated that he can supply the Mizuno heads. Bob believes that Ruger would supply two of their heads. The choice has yet to be made.
• The head's hosel must accept a shaft with a 0.335" tip diameter.
  
• The heads should be in the 300cc volume range. It is desirable for test purposes that they not be too large or forgiving, in order to emphasize any effect that spine might have on off-center hits.
  
• Identical head weights, within one gram.
• The loft and lie of the two heads must be the same to within one degree.
2. Identical length and similar swingweight.
• The length of the clubs will be as nearly the same as we can make them, and will be within 1/4" of 45".
  
• The swingweight will be between D-0 and D-3.
• The foregoing requirements will determine the shaft characteristics.
• It is expected that the swingweights will be very similar, given the control over the parameters that control swingweight. But we will not do anything (like adding weight to a head) to make the swingweights identical.
  
3. Identical shafts except for the magnitude of the spine.
• We will choose a shaft model well-suited to a 100-mph clubhead speed with "normal" tempo and acceleration.
• We will choose a shaft model that tends to the tip-flexible, with a torque rating in the 3.5-4° range.
  
• The shafts will have the same weight within two grams.
  
• See below for how we will characterize the spine.
4. We will measure the magnitude of the spine as the difference between the highest and lowest frequencies as the shaft is rotated in a frequency meter clamp.
• All measurements will be done with the same frequency meter, clamp, and tip weight.
• The clamp will be a 5" clamp, attached to the raw shaft with no grip. The same clamp mounted to the same bench or other structure will be used for all measurements.
• The tip weight will be a 205 gram weight. The same weight will be used for all measurements.
• While frequency will be our primary measurement of the size of the spine, we will (if it is not too much trouble) augment these measurements with alternative characterizations, for instance:
• John Kaufman has used a spin indexer and digital scale to characterize spine.
• The Apache Multi-Match machine can be used to characterize spine.
  
5. We will select four shafts from a larger number of raw shafts. The criteria will be:
• The same nominal frequency to within 5cpm. (We need to determine what "nominal" means. The stiffest frequency? The softest frequency? The average of the two? Something else?)
• Spine magnitudes of 10cpm ±1cpm for two "large-spine" shafts and 5cpm ±1cpm for the two "small-spine" shafts.
• The shaft shows Flat Line Oscillation (FLO) in the directions of stiffest and softest frequencies.
  
• No more than 1/16 inch measurable bend in the shaft.
• The shafts exhibit FLO in the same planes as the stiff and soft sides as determined by a bearing-based spine finder or deflection board. This is to defuse the argument about whether it is better to use "spine" or FLO. We will select shafts for which this is not an issue.
6. We will have the shafts PUREd (the proprietary process that SST uses to determine spine).
7. We will build the clubs using one large-spine shaft and one small-spine shaft.
• In each case, we will use the shaft where the spine positions obtained by PUREing, FLOing, and spine-finding are the most nearly identical. If there is more than a 10 degree difference, we must select more shafts and repeat. (Don Johnson has looked at a small sample of shafts, to compare the various ways to find an orientation "mark" for the shaft. We will review his data to see if this is like to be an achievable goal.)
• We will use the position found by PUREing to build the clubs. (That does not imply that we will be using their recommended alignment of "12 o'clock" as opposed to "9 o'clock". This issue is discussed below.)
8. We will align the large-spine club in the "best" position initially, and realign it during the testing. We will align the small-spine shaft to the "worst" position initially, and realign it during the testing.
• The strong, weak, and PUREd positions will be marked on the shaft in a visible spot, before the primary mark is obscured by adding the grip. We will need this for realignment on the day of the testing. (Golf Labs has suggested that club tests are most instructive with no grip. If we adopt this suggestion, then this point will be modified or dropped.)
• The top of the hosel will be marked with lines at 0°, 45°, and 90° to the clubface, to facilitate alignment.
  
• The team will have to reach some consensus about what the "best" position is, for purposes of this test. If we have an opportunity to do five tests, we will test both candidates with the large-spine club. Otherwise, we will have to choose either S-plane or N-plane alignment. (I have a slight preference for N-plane alignment, but am quite prepared to defer to others with more experience in this.)
9. The clubs will be made to survive the morning of testing, not a lifetime of use on the golf course. This implies a few departures from the usual conservative construction practice, in favor of easy modification during the tests:
• No ferrule will be used. We will be sure that the cone in the hosel is filled with epoxy that "beads" at the top of the hosel.
• We will use a very fast-set epoxy.

7.2. Day Of Test

At least one member of the team, and preferably two or three, should be present at Golf Laboratories for the robot testing. Here is the availability of each team member:

• Alan intends to be there.
• Bob has not indicated an intention.
• Don would like to be there, but doubts that calendar and transportation will allow.
• John would like to be there, pending transportation and calendar. There is also a good chance he can join Alan on the exploratory visit April 22.
• Dave would like to be there, pending transportation and calendar. 

7.2.1. Plan A - If we are able to do five tests

Tests will be conducted on five clubs, the two we have built and three others that we modify for spine alignment on-site (during the testing). The time-line will be something like this:

In the robot, being tested	On the bench, being modified
Large spine, N-plane alignment
Small spine, worst alignment	Large spine, modifying to S-plane alignment
Large spine, S-plane alignment	Small spine, modifying to best alignment
Small spine, best alignment	Large spine, modifying to worst alignment
Large spine, worst alignment

7.2.2. Plan B - If we are only able to do four tests

Tests will be conducted on four clubs, the two we have built and two others that we modify for spine alignment on-site (during the testing). The time-line will be something like this:

In the robot, being tested	On the bench, being modified
Large spine, best alignment
Small spine, worst alignment	Large spine, modifying to worst alignment
Large spine, worst alignment	Small spine, modifying to best alignment
Small spine, best alignment

We will use N-plane alignment for the "best alignment" case.

7.2.3. Common to both plan A and plan B

In order for this to work, we will need to build the clubs (at least the field-modified clubs) with quick-setting epoxy. We will need something that can be hit 15 minutes after it is applied. (This assumes we have 25-30 minutes for the modifications. Mike or Alan, how much time do we actually have while the robot hits 20 balls?) Tools and supplies we will require at the site include:

• Heat source for removing clubhead from shaft, such as torch, micro-torch, or heat gun. This choice will be made by the clubmaker on-site the day of the test.
• Shaft puller.
• Tool[s] to re-prep shaft.
• Tool[s] to re-prep hosel bore.
• Quick-setting epoxy. The choice will be made by the clubmaker on-site the day of the test. The requirement is that clubs can be hit 15 minutes after application. Candidates believed to meet the requirements include:
• Tour Van quick-setting epoxy.
• Epibond 1217 - A/B produced by Vantico.
  
• Club ruler. (We will want to recheck the modified club. Any other rechecking, such as swingweight or frequency, will wait until post-testing.)

The robot will be adjusted for test conditions of:

1. 90 mph clubhead speed for the first club. We will depend on maintaining the same robot settings for subsequent clubs. (See Notes.)
2. Clubface square to the clubhead's path of travel. This will be assured for each club.
3. Impact on the center of the clubface. This will be assured for each club. (Golf Labs does this routinely if the test calls for it; we do not need to supply impact stickers unless we intend to measure impact variation -- which I suspect will be difficult to do quantitatively. If we are interested in this measurement, we need to ask Golf Labs how they do it -- if they do it.)
4. Other parameters supported by Golf Labs will be set for "low handicap". (Golf Labs claims to be able to simulate high, medium, and low handicap swings. We will have to decide which one we want. My inclination is "low", but the subject is certainly open for discussion.)
   

All tests will be run with:

1. Identical balls, to the degree that Golf Labs can supply them.
2. Identical robot parameters controlling the swing profile.

For each ball hit, the following will be recorded:

1. Carry distance and direction/dispersion. (Note that, on Golf Laboratories' data sheets, they refer to the directional error in feet as "dispersion".)
2. Total distance and direction/dispersion.
3. Ball speed.
4. Clubhead speed.
5. Launch Angle.
6. Initial spin rate.
7. Wind speed and direction relative to the hit. (We hope this will be uniformly zero; we intend to select a time slot when no wind is the norm. However, if that does not occur, we need to know which hits were corrupted by wind.)
   

Golf Laboratories routinely includes statistical summaries and graphs of the measurements. These will most definitely be helpful, but the raw data above is essential.

7.3. Post-test activities

After the day of the test, the activities will consist of:

• Rechecking the clubs 
• Make sure the modified clubs have swingweight and frequency consistent with what we expected.
• Reducing the data
• Copies of the raw data sheets from Golf Labs will be distributed to the team members.
  
• The statistical summaries that we get from Golf Laboratories should minimize this task.
• Analyzing the data and drawing conclusions
1. First let's each eyeball the data and bring to bear our favorite methods of analysis.
2. We will share our observations and conclusions.
3. We will discuss to achieve a consensus, pending a report.
• If we agree on the conclusions, then the report will be fairly easy to write.
• If we disagree on a few specific points, then the report will mention the areas of disagreement and arguments for each interpretation.
  
• If we fail to achieve consensus on the conclusions, the report will be in the form of a single document with alternative, signed conclusions sections.
4. Things we should all agree to about reporting our results:
• There will be a report. No matter what the results, whether they agree with our preconceptions, whether we consider the tests valid or flawed, we will produce a report that we can make public. The report should contain our reservations, but this work will not just "vanish into the ether".
• After the tests, we will agree on a timetable for the preparation of the report. No public statement of results will be made by a team member until the report is made public. If the report is delayed by more than two weeks from its scheduled date, then public discussion may begin; this is to prevent using the lack of a report to consign it to permanent secrecy.
  

Notes

1. Required number of hits: Assuming that the shot dispersion is relatively gaussian, this is going to be the tradeoff between enough hits with one configuration to get the standard deviation good enough to be able to discern variations due to changes in configuration. This is a bit of a Catch 22. We don't know how many shots to take because we don't know what the changes will be. I suspect that we are going to have to decide what constitutes a 'significant' variation and design our tests to assure that we can resolve that level of variation with some level of confidence, if it occurs. I am hoping that this is something that Golf Labs can help us with. One of the things we don't know at this point is what variation in shot dispersion can we expect from the same club hit multiple times on a dead calm day at constant temperature, for instance. It is clear that changes in shot dispersion statistics is going to be the criteria for identifying shaft spine variations and so it is going to be essential that we understand everything we can about what affects shot dispersion.
2. Order of testing alignments: We want to test A1 then B2 then A2 and then B1. We may not want to put both worst spine alignments later in the morning. It is unlikely that we are going to get perfectly calm winds for the duration of the tests (if at all). At best, we will be getting temperature variations in the air over the polo grounds. That means that we are going to have to factor into our data the effects of temperature, and wind velocity and direction on shot dispersion. This is information that I would expect Golf Labs to have available for us. The inherent dispersion is likely to change over the course of the testing. Distance obviously will (with wind velocity if nothing else). This will affect the confidence with which we can resolve differences due to a given magnitude of change due to configuration changes. We are going to have to decide if we want the best confidence in resolving differences for spine alignment in a single shaft (in which case we would likely test shaft A, for instance, in two alignments early when the winds are calmest and then shaft B after the winds start picking up) or whether we want best resolution between smaller and larger spines.
3. Robot adjustments between clubs: The robot uses a torque motor to drive the swing and controls current to the motor.  In principle, shaft stiffness will have a small effect.  Changing the shaft stiffness without changing the robot settings can result in a slightly different club head orientation at impact, this will change direction and distance.  The issue is whether we want to control distance and direction by tweaking the robot parameters or leave the robot parameters fixed and accept the distance and direction variations.  Alan Brooks recommends leaving the robot parameters fixed because trying to match distance and direction would burn up a lot of shots and we can only get it approximately the same anyway.  The repeatability of the robot is probably better than our ability to tweak the parameters to get a repeat of distance and direction.
   We currently have an issue whether the absolute distance and direction of the shot is important, or only the variation (dispersion). We will have to resolve this as part of the test planning, as it may impact the robot adjustment strategy and will probably impact the number of hits needed for each test.
   One of the things Alan intends to discuss with Golf Labs is what club parameters they have found most affect shot dispersion and then we want to try and minimize all of those except for shaft orientation.
   
4. What to do with additional tests: There are a few very desirable things we might want to do if we can test five clubs, not just four. Given the opportunity, we will have to decide which to choose, among:
• The current plan of record (Draft 1.2) uses the extra test to see which of the two advocated alignments (S-plane or N-plane alignment) is actually best.
• It would be very useful if we could repeat A1 or B1 at the end of the testing. Being able to repeat the results of earlier tests after two shaft changes would give us confidence in our control over test parameters. If the winds are different, it would provide validation of the dispersion versus wind velocity and direction relationship we get from Golf Labs.
• A team member has suggested doing a test on a club with a zero-cpm spine. This might begin to quantify the "supershaft" effect, though matching the two shafts in frequency is a challenge since a high-spine shaft has several interesting frequencies and we really don't know which to match.