2nd Gen Regular v Mid-Grade v Premium Fuel Experiment

Let's say its Monday and I have a full tank. Come Friday my odo says 300 miles have been travel since Monday. Friday I fill up at the gas station. The pump clicks off and the tank takes 15 gallons. So in this scenario I got 20mpg. Now lets say I fill past the click and go to 15.5 gallons, I would get 19.35 mpg.

Not sure where the tank to tank estimate would go wrong if you are using the exact amount that the pump read ( even with over or underfill, the gas pumps is exact to two decimals) vs the exact amount of miles since last fill up? Its not like the miles driven or the gas dispensed are unknowns, not sure how one would insert errors into either ya know.

Disclaimer: Stats was never my best subject.

 

Exactly, for this individual measurement you do not actually know how much fuel you burned for this tank you only know how much you put in which should be close to what you burned but is actually slightly different. The difference between the actual amount burned and the amount you put in the tank depends on two things actually, how much the tank was over or under filled the last time you filled and how much it was over or under filled this time.

Since what you put into the tank at the gas station is not actually the exact amount you burned out of the tank for your odometer miles you introduce an error in the MPG estimate for that tank fill up. In this case you over estimated the amount of fuel burned because of the overfill and thus under estimated the MPG.

Now the key point here is that on the very next tank if the pump "properly" fills the tank tank and doesn't overfill it you will now under estimate the fuel burned (you actually burned 0.5 gal more) and thus over estimate the MPG. In this case the two errors perfectly offset each other and if you measure the MPG across the two tank fills (600 miles/30 gal) you'd get exactly 20 MPG. Of course the reality is probably the errors across two tanks don't perfectly offset, each individual tank fill with be a bit over or a bit under. So even if my truck is burning 20 MPG constantly if I look at MPG estimates of (odometer miles)/(gallons to fill tank) for every tank they won't be 20 MPG every fill.

Now, it is quite easy to reduce this error by measuring over multiple tanks as the OP has done. So continuing your example say we drive 300 miles for each tank and we do this for 10 tanks (so 3000 miles). And say that the engine is in fact running at exactly 20 MPG all the time. And further say we still end up with a 0.5 gallon overfill by the end of our run of ten tanks. To compute our MPG we'd then have 3000 miles driven (we know this exactly thanks to the odometer) and we think we've burned 150.5 gallons (but we are a bit wrong, we've actually only burned 150 gallons). Now the error is quite small, 19.93 MPG computed vs the 20 MPG actual.

So again, the gas dispensed is not unknown, but the gas dispensed is not the gas burned for the mileage you took off the odometer. However, over the long term if we measure across many, many tanks and compute MPG as (total odometer miles for all tanks/total gallons from all fills) then we can get a very accurate estimate of the true average MPG that is minimally affected by the overfills and underfills. Effectively the long run over and underfills cancel each other. Please also note that the above estimate of MPG as total miles over total gallons is in fact different from taking the individual MPG estimates for every tank and averaging those together. The former is a better estimate than the latter it turns out.

Anyway, the point I was making in my post was that the way the individual tank MPG estimates average out over all the tanks measured is different for the tank fill up error than it is for driving conditions variations. Specifically the tank fill up errors average out much faster for a given number of tanks than the driving conditions variations do. So if you look at the standard deviation of all the individual tank MPG estimates you will be seeing them vary both because of tank fill up errors (which largely go away in the average) and driving condition variations (which remain to a much larger degree for a given number of tanks measured).

Again, sorry for the diversion - but I guess I was the one who started it so I'll try to clear it up too...

Nor mine!

 

Thanks for the write up. Your explanation sounds reasonable. When I was thinking about this yesterday I was trying to understand when it would be better to lump a bunch of individual measurements into one huge measurement. I hadn't considered your point.

I believe you when you say a major component of the error is in the pump cutoff point. Do you have any evidence that it is in fact a major component?

 

Thanks for the writeup. I'm just not sold on the pump cutoff. If the pump says it dispensed 15 gallons, are you saying that sometimes it may actually pump 14.5 or say 15.5, either a little less or a little more. This doesnt seem to add up to me.

 

Well I should probably have been more specific. The pump cut off can be a significant component of the error for an individual tank measurement, but how much depends on how empty you let the tank get (how much you have to fill up) and how bad the pump is at consistently cutting off. The larger issue though is that the degree to which it can corrupt the significance of the standard deviation of individual tank MPGs is potentially high.

As an example, you probably know from personal experience that a typical variation of at least 0.2 gal or so is reasonable. But if you really just let the auto cut off do its job and never top off manually it is not unusual to get it clicking off a full 0.5 gal or more "early" from time to time.

So that means we'd expect to frequently get 3% or slight more in error just from the pump cutoff. Not huge to be sure. The issue is how is this error reduced over multiple tanks compared to a random and uncorrelated variable.

Let's say we measure over 100 tanks. The expected error from pump cut off will be just 0.03% at that point (just 3%/100 in this example). This is because of how we do the averaged measurement (total miles/total gallons). The expected error in "total gallons" is still just 0.2 to 0.5 gallons, the error of single "pump cutoff error". The "pump cutoff error" does not accumulate to larger values over multiple tanks in this measurement.

Thinking of it another way, if you end up under estimating the MPG for a tank because the tank was overfilled you are more likely on the next tank to over estimate the MPG. The estimate errors are correlated, they are not independent. The probability of over estimate on the second tank is dependent on whether the prior tank was an over estimate or under estimate. When I flip a coin and get heads for the very next flip the probability of tails is still exactly 1/2, it does not depend on the result of the previous coin flip at all. The statistical significance test you'd make from a standard deviation is for independent variables like the coin, but in this case part of our error is from a dependent variable from our inconsistent tank filling.

Now consider the behavior of a random and uncorrelated error - this is the type of error that a statistical significance test over an ensemble of samples is designed to evaluate like the coin above. In this case the reduction in expected error over multiple trials is reduced by the square root of the number of trials. So for our 100 tank example if we had a truly random and uncorrelated variable that gave us a 3% error then after 100 tanks the expected error would only drop to 0.3% rather than the 0.03% for our tank filling issue.

So the real question as to how significant is the tank filling error is really how big is the tank filling error compared to the variation in actual MPG because of driving conditions and what not. If the driving conditions variation for a single tank is say 10 times bigger than the tank filling variation then our test statistic will still be dominated by the driving conditions which we are assuming are of the random uncorrelated variety. Yes the tank filling error doesn't average in the same way, but it is so small in proportion to the driving condition variation that we could probably safely ignore that. If however the two errors are nearly the same magnitude for a single tank of gas then our test statistic would be off by just about a factor of two when we average a large number of tanks. And of course if the tank filling error was much greater magnitude than the driving conditions error then we could be off by a huge amount.

Given the numbers and likely variations just based on my driving I'd guess that really worst case the two variations are about the same magnitude. So the tank filling variation is only partially corrupting the result. And so as I said in the first post you could probably just use the test statistic and keep in mind that it is probably a bit pessimistic. And really, that's just fine as that would be the conservative choice for determining statistical significance.

Again, sorry for the nitpicking diversion...

 

No, no. Sorry if I'm not being clear.

Assume if the pump says it gave you 15 gallons that it did in fact give you exactly 15 gallons and that this is always true.

The issue is what the pump gives you and says it gives you is not actually the amount of fuel that you burned.

For example, one day you decide to start tracking your mileage. You go to the gas station and "fill up". What the gas pump says this day is of course irrelevant, you haven't been tracking anything yet. But the point is what is "filled up". Depending on the pump and your pumping technique your tank might now have 20 gallons in it, or perhaps 19.5 or perhaps 20.5. You consider it "full" but you really don't accurately know what "full" is. Let's just say that for whatever reason the pump cut off a bit early and your tank now has 19.5 gallons in it.

So you drive 300 miles. You go to the gas station and the pump tells you that 16 gallons went into the tank. And in fact exactly 16 gallons went into the tank. However, when you got to the gas station your tank actually had 4.5 gallons left in it (you'd burned 15 of the 19.5 gallons that were in it). And this time the gas pump cut off late and filled your tank "full" to 20.5 gallons rather than 19.5 gallons. Your truck actually burned exactly 15 gallons over 300 miles for 20MPG. However, you think based on what you put into the tank to "full" that the truck burned 16 gallons over 300 miles for 18.75MPG. Your MPG measurement is wrong because you used the wrong number of gallons. You used the number the pump told you, and the pump was exactly right about how much went into your tank on this fill. But that isn't how much you actually burned between the fills.

So you see the pump was perfectly accurate in telling you how much fuel went into the tank. The problem is the pump is not filling the tank up to the same level of "full" each time. You are assuming that it does when you do the calculation (miles driven/gas put in tank). The pump is accurately telling you "gas put in tank" the problem is that "gas put in tank" does not in fact equal "gas you burned" over "miles driven".

So the pump is accurate, and no gas is leaking out of your tank or anything. But for each "tank" that you fill up and compute a MPG estimate you are getting an error. You've got the miles dead right from the odometer. And the pump told exactly correctly how much went into your tank. But how much went into your tank is not exactly how much you burned to go that many miles.

Clear as mud?

 

You've made it plain as day!

 

I'd run regular if I could, But being boosted + the smaller S/C pulley I have to run 91+ In my car I run 91+ also but I have a tune in it & even when it was stock I ran 91+ even though I could run 87, It always seemed to run better on 91+ & made that little bit of extra power (factory calmed 310 on 91+ vs. 306 on 87) & seemed to get about 1-2 mpg better. But on the truck, I would deff run 87 if I could!

 

Thanks for the thorough explanation. You seem to know your stuff.

I'm curious to know if it is possible to quantify what the tank filling error is using this data. It seems like it should be straight forward if pump error shrinks as more tanks are averaged, and random error stays the same. Then you can sub-sample the data and compare the sample of single fill ups vs sub-samples of groups of fill-ups. The difference in error should be entirely due to the pump handle error, assuming there are just the two sources of error. is it that simple?

 

Yep, that is the general idea. If you have enough individual samples you can do various curve fittings, histograms and so forth of different sub-groups of the full sample and model the data as the sum of two different errors with different statistical properties. The practical problem usually is not having enough data to do a good job of that. And usually to even start doing that you have to have some sort of starting model of the two error sources/distributions such that all that is left to determine is their relative magnitudes. You quickly end up sort of calculating statistical properties of statistical properties and the number of samples required to do it with any degree of confidence becomes very large.

Notionally for this example you could compute the standard deviation with individual tanks, then the standard deviation with pairs of tanks averaged, then the standard deviation with four tanks averaged and so on. You could then plot the standard deviation as a function of number of tanks averaged, do a curve fit of it and the parameters of that curve fit would give you the relative weights of the tank filling error and the other (assumed independently random) errors.

But of course for a fixed set of measured tanks, say 100, our estimate of the standard deviation for the averages of more tanks gets worse and worse the more tanks are included in each average. For instance if averaging 20 tanks together we'd only have 5 unique data samples after averaging the original series of 100 samples. Computing a standard deviation from just 5 points is going to be a pretty sucky measurement of the standard deviation and thus that will make our curve fit and our estimate of the relative weights of the two types of errors suck as well. And so then we'd want more than 100 fills to work with to get better estimates of the standard deviations for higher averaging and eventually the problem becomes impractical.

 

I am enjoying this minor derail. It's still relevant to the topic in a lot of ways. Other people are probably learning something from your detailed explanations. Thanks for taking the time to write it all out.

I don't quite understand why you made the assumption in the third paragraph about the sample needing to be unique. In this example i don't see why the subsample has to be of contiguous fill-ups, and not overlap. Certainly the pump handle error is correlated between subsequent tanks. It's also true that averaging across more tanks causes the pump handle error to approach zero. It must follow that, in the limit, the pump handle error has a mean value of zero. And if the mean is zero, then a random order sample should work just as well as a continuous one, given the sample is large enough. Assuming 20 is large enough, your sample of 100 tanks turns into a huge number of subsets.

If I have all that right, then based on what you've said in the second paragraph it should be possible to determine the contribution of pump-handle error to total error. I'm not going to ever bother calculating it myself. I don't have the data anyway. But I'm content to know that it's possible.

 

Ha! I was actually considering putting an asterisk on that paragraph to point out you can't use overlapping data sets but thought no one would probably notice or care about the answer. You anticipated that perfectly...

I'd have to go dig out a bunch of nearly inscrutable books to refresh myself enough to give you a very satisfying answer I suspect. But if you'll settle for a short half-assed answer we can focus on just why that wouldn't work for the case of a truly independent random variable - which is the kind of variable most all these significance tests assume and the kind the pump error is quite specifically not. We are assuming the errors beyond the pump error are in fact truly independent though.

If I average samples of an independent variable and make sure the sub-groups don't overlap then the resulting average samples are themselves like that of an independent variable just with a different standard deviation. It is pretty easy to see the averaged samples from non-overlapping sub-groups are independent.

However, if I use overlapping subsets (a running average being an easy example) then clearly the resulting average samples are in fact not independent. As different overlapping averages contain some of the same underlying samples/values from the original data set these overlapping averages are related and have correlation to each other. So we are immediately in a dangerous land where we have created a non-independent variable that we are going to analyze as if it were an independent variable. There be dragons here!

Whether you can/should use overlapping sub-groups in any statistical analysis or study is very much a "it depends" kind of thing and also a place full of deep pitfalls. A classic example of this is testing models of markets like the stock market against historical data. We only have so much reliable historical data and so the temptation to massage the crap out of it to get more sample periods to test against is very strong. In many cases for historical market data using overlapping samples is definitely the wrong thing to do for an analysis and can lead to erroneous results and overconfidence in a model that it turns out is actually just a random anomaly given the wrong statistical confidence. However, for other kinds of historical market tests in fact using overlapping data is absolutely the correct thing to do!

Sorry for the muddled reply, but its the best I've got for you at the moment.

Yes, it is definitely possible. And in fact there are techniques for one to hypothesize any particular probability distribution function for a set of data and then come up with a level of confidence that a given set of samples came from a system/process/variable with that probability distribution. But that is often very tricky to do right and usually requires lots and lots of data to end up with significant/accurate results. What you are proposing (determining the relative contributions of the pump error to other errors) is basically a simpler and more straightforward special case of that more general problem.

 

Clearly using overlapping samples is a tricky topic. It makes sense that you'd run into issues when the underlying distribution has any kind of correlation between measurements. Again, thanks for the detailed explanation.

 

This post has clearly turned into a discussion for statistical brainiacs which I have found interesting and numbing. For most of us, the OP's long term analysis of the amount of fuel purchased and miles driven, is sufficient to illustrate the MPG difference between fuel grades. Not withstanding statistical anomalies, I compute my MPG by dividing the number of miles driven, by the amount of fuel the pump registers went into the tank. I'm good with that and the OP's data. I'll continue to use 87 octane.

 

Exactly, most all my posts in this thread are just a statistics sideshow/thread derail. The larger point is the OP did an excellent and controlled test to show what we mostly all knew already - 87 will get you the same mileage as any other grade (assuming same ethanol content in all grades). Great to see someone do such a long running test and share the results.

 

With it already being spring, giving this thread a quick rez as there are probably drivers thinking about summer driving.

 

On the over/under fill :
I have added up to 2.5 gallons from when the pump first shut off. Let's save the arguments for NOT doing this for another thread (I only do this on a road trip where I will be driving enough to burn it in an hour or so.)

I have also noticed that how soon a given pump "clicks off" seems to vary quite a bit from pump to pump, and to some degree how busy the station (# of pumps in use while I'm filling up).

Measuring multiple tanks as the OP did is the only way I can think of to reduce the error.

Thanks again, OP.

 

I think most will go with the initial shutoff (as do I) as a reasonable point to measure from. When I was really keeping an eye on mileage, I could never be sure how much more you could squeeze into a tank with certain pumps, so I just began calling it full when the pump clicks off, and have stayed with that since...if you do it the same every time, there can't be that much variance. I also use the same few stations almost all the time.

Very enlightening read, however, on the experiment, thanks OP. Had to skip over the side track on statistics, though.

And we're measuring mpg on a truck because...?