Friday, January 28, 2005

Marcel ZiPS alongside PECOTA, or How I Learned To Quit Worrying and Love Non-Humorous Half-Pun Titles

By request of Tangotiger by way of Jon Weisman, I’m delving into three sets of offensive projections for the 2005 Dodgers to see how they stack up. The projection systems in question are Baseball Prospectus’ PECOTA (subscribers only), Dan Szymborski’s ZiPS, and Tangotiger’s Marcel.

Projection systems have a bit of a Rorschach effect; the goals and methods depend on whomsoever produces them, and the results can therefore vary substantially in even very similar systems. The three systems in question here use a very similar process: take the player’s recent performance numbers and regress them. Simple enough, but the question of what population to regress toward varies substantially. In other words, what population (slap-hitting 5’7” guys, power hitting 6’2” guys, and so forth) players are grouped in for regression analysis differs greatly depending on the system. PECOTA’s at one end of the spectrum, putting a great deal of effort into finding a population of similar players for comparison, and Marcel is at the other end of the spectrum, using extremely basic population definitions.

Each approach has its merits. Better defining the population a player belongs to will yield a better projection. Taking the extra steps to do so, though, is often statistically problematic, as defining a player as part of a population on the basis of his performance record creates something of a cross-fertilization effect. For any group of players who are, from a true talent standpoint, very similar, some will have underperformed over a period of time and others will have overperformed. Thus, looking only at those players who hit one home run every 20 at bats over a three-year period doesn’t necessarily define the true population of a player who has hit one home run every 20 at bats over the past three seasons. Thus, there’s certainly an accuracy argument to be made for systems which privilege elegance and simplicity; the relative minimalism of Marcel and to ZiPSs is useful for reasons beyond their accessibility.

Throw on top of those concerns the highly contested world of statistic translations – different applications of park factors, different calculations for minor league equivalencies, and so forth – and you should expect a substantial amount of noise separating different projections. In other words, these projections, like bland soup, come with several mandatory shakes of salt.

Taking a look at the Dodgers, it’s pretty easy to catch a few general trends in the differences between the projection systems. For example, Marcel tends toward higher batting averages and lesser secondary production from most players than ZiPS or PECOTA.

Furthermore, how we compare projections can be somewhat problematic. The HBP, SF, SH, and IBB aren’t counted in the data that Baseball Prospectus publishes, and ZiPS is also SH and SF deficient. To convert the numbers into runs, then, I can either re-run all the numbers to give myself a guess of how many SF/SH/IBB etc. each system is working with, or I can use a blanket system which pretends that those events don’t exist. To maintain my admittedly limited sanity, I opted for the latter, using a modified linear weights system (which I have assonantly and perhaps asininely termed “Fake Weights”) to compare the projections. I could get into the formula here, but it’s probably of little interest to anyone; I’ll just say that I compared it with straight linear weights for the Marcels and the MLVR’s for the PECOTA’s to ensure it was about right. For each individual player, it shouldn’t really impact the differences between projections. The Fake Weights are not position adjusted.

For each player, I’ve listed their high and low projections with Fake Weights and Gross Production Average, as well as the average of the three systems. Fake Weights are prorated to 620 plate appearances since, well, playing time projections are worth roughly what the vice presidency was worth to Cactus Jack Garner. Also, try not to freak out by how low these look; they’re not park adjusted and they’re based on Dodger Stadium (or whatever statistical model of it the various projection authors chose to use; if I recall correctly, ZiPS and PECOTA both made minor ad hoc adjustments for the renovation). Also, I only looked at current Dodgers who appear on all three projection systems and have some degree of likelihood to make the club to start the season. Cody Ross, Henri Stanley, Chin-Feng Chen, and Brian Myrow were all left off one or more of the published datasets, and I didn’t get around to DFA survivor Joe Thurston.

J.D. Drew
High: ZiPS (+34.1, .312). Low: Marcel (+27.5, .303)
Average: .285/.395/.518, +30, .307
Marcel and PECOTA were roughly the same for Drew; Marcel gives him a slightly higher batting average while PECOTA assigns slightly more walks and power. ZiPS bets the over in all three categories.

Hee Seop Choi
High: PECOTA (+18.3, .292). Low: Marcel (+8.4, .274)
Average: .253/.367/.465, +12.3, .281
Choi is the kind of player for whom we can expect PECOTA to be higher than Marcel and ZiPS, since his power thus far in his major league career has been strangely absent (relatively speaking) despite having the minor league pedigree and the scoutish projectability. The only major difference among the projections is that PECOTA’s high on his power. I’d probably take the over on that +12.3.

Jeff Kent
High: Marcel (+15.1, .284). Low: ZiPS (+7, .274)
Average: .280/.342/.498, +10.6, .278
Marcel bets the over, ZiPS the under, and PECOTA’s .276/.341/.498 approximates the average.

Milton Bradley
High: ZiPS (+11.6, .281). Low: PECOTA (+7.5, .275)
Average: .278/.370/.447, +10, .278
Not a lot of spread here. Marcel digs his power, ZiPS digs his walks. No real substantive differences.

Jayson Werth
High: Marcel (+4.5, .269). Low: ZiPS (+0.4, .265)
Average: .257/.336/.465, +3, .267
Marcel likes his batting average and not his walks, PECOTA likes his power and not his batting average, and ZiPS only likes his walks. Not a big spread here.

Olmedo Saenz
High: ZiPS (+1.9, .271). Low: PECOTA (-9.2, .258)
Average: .259/.341/.446, -2.2, .265
PECOTA digs his walks but sees a batting average collapse (.244).

Ricky Ledee
High: Marcel (-0.9, .260). Low: PECOTA (-6.9, .249)
Average: .236/.331/.426, -3.8, .255
PECOTA’s not a fan of the Dodger’s older role players, jeering Ledee’s average and power. ZiPS strongly resembles that average line.

Mike Rose
High: Marcel (-3, .258). Low: ZiPS (-4.9, .255)
Average: .254/.349/.395, -4.2, .256
It’s tough to get these projection systems to agree more on one player’s offensive value, although they disagree substantially on how he’ll get there. Marcel is high on his batting average and power but low on his walks (.267/.333/.433), while the other two expect his walks to keep up but PECOTA says yes power no average and ZiPS says yes average no power. Cherry-picking the worst components yields .242/.309/.360; cherry-picking the best yields .267/.378/.433.

Antonio Perez
High: Marcel (-0.6, .260). Low: PECOTA (-11.9, .248)
Average: .260/.337/.406, -7.8, .253
If you want to wager on his rate stats, bet the over since he’ll see a ton more left-handed pitchers per PA than this data assumes. ZiPS roughly matches PECOTA on this one.

Jason Grabowski
High: ZiPS (-7.3, .253). Low: Marcel (-12.6, .245)
Average: .245/.325/.410, -9.4, .249
Only substantial difference here: Marcel doesn’t care for Grabowski’s secondary skills.

Jose Valentin
High: PECOTA (-6.7, .252). Low: ZiPS (-25, .229)
Average: .226/.297/.441, -13.5, .244
PECOTA (.235/.312/.445) and Marcel (.233/.301/.460) pretty much agree on him, with PECOTA more optimistic on his walks and Marcel more optimistic on his power. ZiPS doesn’t share PECOTA’s discipline optimism or Marcel’s power optimism and throws a skunky .209 batting average into the mix. This dish may need even more salt, since Valentin faced southpaws much more in 2004 than he will in 2005 and because his overall offensive value will likely outstrip his raw totals since he’s shown outstanding situational hitting tendencies.

Dioner Navarro
High: Marcel (+6.5, 271). Low: ZiPS (-25.4, .229)
Average: .264/.326/.387, -14, .243
Marcel (.286/.353/.451) seems to have messed this one up, and I’m tempted to say a data error (perhaps deriving from his wildly successful cup of coffee in September) is at fault. ZiPS pegs him at –23.1, so Marcel is clearly the outlier. ZiPS likes his average and not his power while PECOTA likes the power and not the average. It’s hard to imagine the Dodgers will burn a year of his service time this season.

Dave Ross
High: Marcel (-14.1, .235). Low: ZiPS (-22.6, .223)
Average: .228/.311/.410, -17.3, .242
Marcel likes his batting average, PECOTA likes his secondary average, and ZiPS likes neither.

Cesar Izturis
High: ZiPS (-18.6, .235). Low: PECOTA (-25.4, .225)
Average: .273/.312/.361, -21.1, .231
PECOTA, as has become a theme, is higher on his walks than the other two, but it doesn’t think much of his batting average (.261). ZiPS dislikes his “power” but sees him keeping most of his watershed batting average (.284). Marcel splits the difference on average, kind of digs the “power,” and is pessimistic on the walks.

Paul Bako
High: Marcel (-26.7, .224). Low: ZiPS (-41.3, .205)
Average: .224/.299/.321, -34.7, .215
His best projection is lower than anybody else’s worst projection, although it’s close. Kind of like Jose Valentin if you take away all of the home runs. He’s pretty much replacement level with the bat for a catcher, so if he’s about a run per week better than Ross behind the plate he makes sense. I’m not touching evaluations of catchers’ defense with a pole of any reasonable length, so I won’t engage in any conjecture.

Tom, great work!

I just want to clarify the Navarro situation: I have his reliability as .03 (the higher the better). In this case, there are 782 players with more reliability than Navarro. A reliability of .03 means that I regress their (limited) performance 97% towards the league mean. Essentially, Marcel marks all rookies as league average hitters. Clearly this is wrong. But, not so clearly if this is really a big problem.

The more PAs a rookie gets, the more likely he'll hit above average. The less PAs, the more likely he stunk up the joint. But, at the end of the year, when you validate the forecasts, you typically apply a min PA cutoff, something like 250... and lo and behold, you won't find many rookies who stunk up the joint for 250 PAs.

Choi on the other hand is interesting. I regressed his performance 27% towards the mean (reliability of .73). That's about the same as Junior, with about the same overall performance. But Choi also has alot of minor league data that Marcel knows nothing about.

These are the kinds of players that Marcel is no good for. Marcel purposely discards this information, while other forecasters embrace them. The other forecasters should really stand out here.

Seems to me that the interesting comparisons among forecasters are those:
- with less than 700 MLB PA in the last 3 years
- anyone that's 24 or younger or 36 or older
- changed types of parks (like Larry Walker)

All the other players looks like there's not much difference, and really adds noise more than anything, in trying to evaluate the systems.

Again, I appreciate the review!
Is there a system that can tell us: A) Are the Dodgers a better team this year that the starting lineup last year? and B) Will the Dodgers win the Division this year, and if not, who will?
Stephen Bright
Those are certainly things you can attempt to project. The degree of accuracy is, of course, the key issue. All three of these projection systems also have pitching projections, so if you want to roughly model how many games the Dodgers will win you can take these offensive projections and assign playing time to each player to get runs above average. You can then take the pitching projections and do the same thing, though I'd suggest running the K/BF, BB/BF, and HR/BF numbers into a projection and then determining how many singles, doubles, and triples will be yielded by combining each pitcher's batted ball type rates with whatever fielding projections you prefer. In all, it's a lot of work, and if it's something you want to do than more power to you. I'm not too sure it's something I'd like to do, since the amount of work it takes doesn't match up to what the final product tells us in the long run.

Every year Diamond Mind Baseball runs 100 full simulations of the season based on their projections, and that's normally a decent measuring stick. Those come out a little before opening day.

At the end of the day, though, the Dodgers are, to me, clearly the favorites in the division, and their offense should be as good as or better than was last season's edition.
My take on how they finish in the West this year Records included:

1. San Diego 94-68
2. San Francisco 91-71
3. Arizona 90-72
4. Los Angeles 82-80
5. Colorado 77-85

That's the way I see it and I'm sticking with it. And just to think I used to be such an optimist.....
If that's how you "see" it, I take it, then, that we can term yours the "eyeball" methodology. What is your confidence interval?
Hi Tom, and thanks for your thoughtful answer. I, too, feel the Dodgers are better off this season, both in offence and in pitching.

Somehow, I have the image that Depodesta has a laptop program that instantly calculates the Dodgers' record with each new signing, and that he knows exactly the number of runs needed to win the Division. Have you heard of him hiring a Sabermetrics guy to help him with that? Or maybe he's just got someone to write the programs for his laptop.

My problem is I've felt very confident about our chances every year, EXCEPT last year!
Stephen Bright
Your blog I found to be very interesting!
I just came across your blog and wanted to
drop you a note telling you how impressed I was with
the information you have posted here.
I have a average salary of professional athletes
Come and check it out if you get time :-)
Best regards!
Don't you love me

You can see me here

[url=]My Profile[/url]
[... ] is other useful source of tips on this topic[...]
Genial brief and this fill someone in on helped me alot in my college assignement. Gratefulness you for your information.
i really adore your writing taste, very exciting.
don't quit as well as keep creating mainly because it simply truly worth to read it.
excited to look into far more of your current content, regards ;)
hi there, quality webpage
the simple way to get blog posts that will smash in your visitors:
То столовать человек употребляет алкоголь (вдруг очень дешевый из наркотиков) после, для вновь же губить от своей реальной жизни, которая ему чем-то неприятна. Чем именно - разбираться недосужно, проще выпить. Таким образом он прячется через своих проблем одинокий некогда, два, три… А дальше появляется стращание возникновения наркомании по алкоголю. Многозначащий признак начавшейся алкогольной болезни - одинаковый дискомфорт для трезвую голову. [url=]ремонт 1 квартиры[/url]
Post a Comment

<< Home

This page is powered by Blogger. Isn't yours?