I was only commenting on the observed shifts in your data because you gave the fit and the noise-less data. As for my fit, I did absolutely NO shifting to fit the original noise-less data.
Also, look at your first NNA plot. At an X_value of 50 your estimate looks like Y_estimate~15. But all the data from X>16 is all above 15. Clearly you are doing something wrong.

Let me guess. In your simple test plots near the top, your "multiple times" was about 100, right? Since your averaging window 4 is even, you will get a 1/2 index shift each time you apply it. Your shift is about 50 so I figure you iterated about 100 times. Try your NNA's with an odd number so that you are averaging the target position and an equal # on either side and your pronounced shifts should go away (you can still obviously get some random shifts, but they should be much much smaller that the rediculous ones you have now).
Note: if you are using much more than 100, then you might be doing things better than I state above, but it is still clear you have an algorithm induced error.

Boy, that is some kind-o-crappy data. It brings to mind that there is no magic perfect filter for this kind of application where the actual values can have a relatively arbitrary form. You are basically stuck with a trade off between trying to reduce the noise and to preserve what you can out of the higher spatial frequencies.

I am not sure what you mean by 'have the tendancy to push your points "up" in the direction that you iterate' but it seems obvious to me that you have an artificial ~450rpm shift in torque because of the way you are filtering. Since HP is proportional to TQ*rpm that MIGHT account for the HP shift, I can't tell. Again I urge you to use an odd # of samples in you NNA so that the averaging window stays centered on the target.

Also, your use of very small NNA windows and then re-using it many many many times seems counter productive to me. The width of the impulse response to such filtering is 1+(NW-1)*N_iteration (where NW is the width of your averaging window per smoothing). That is why the fit "appears to keep going after power is pulled" and you won't have very good ability to preserve any high to moderately high spatial frequency content. You would be better off using something like a single iteration of something like the "suggested filter window" shown in that attached plot (although you need to pick its 2 widths yourself). Yeah, the resulting plot won't look as smooth as yours do, but on the one hand you complain about lack of high spatial frequency responce
if on the other hand you want plots as smooth as yours in the presence of the kind of noise you have... well good luck. Sorry, but you can't have it both ways.

Alternatively, you can take a hint from the image process world and use a median filter or combine a median with one of a variety smoothing filters. The median will get rid of the bad "high flyers" that are obviously bogus and then the smoothing filters don't have to work so hard and can preserve more of the higher spatial frequencies.

Good luck,
B

 

Wow, this is INTERESTING. I merely have a minor in Mathematics (major Computer Science), but I follow what is being discussed. I wish I could contribute something but these last couple of posts reached my limit. You guys are obviously serious about curve-accuracy! I'll ask my savvy friends if they got any ideas...

Jeremy

 

That's not clear to me. If if my reasoning is incorrect I definitely want to know why.

An good guess, but incorrect. I'm only applying the filter 4 times (which a window size of 4).

Quite likely, although I _am_ using an equal number of points left and right.

EDIT: yup, you're right. One big glaring bug had me not using equal length windows on both side and that shifted everything WAY right.

[quote]

Boy, that is some kind-o-crappy data. It brings to mind that there is no magic perfect filter for this kind of application where the actual values can have a relatively arbitrary form. You are basically stuck with a trade off between trying to reduce the noise and to preserve what you can out of the higher spatial frequencies.
[quote]

Exactly. It's also the data of data where you want to be able to see very quick and severe dips as they can correspond to engine events.

Me too, but again, I'm not applying it a high number of times.

[quote]

The width of the impulse response to such filtering is 1+(NW-1)*N_iteration (where NW is the width of your averaging window per smoothing). That is why the fit "appears to keep going after power is pulled" and you won't have very good ability to preserve any high to moderately high spatial frequency content. You would be better off using something like a single iteration of something like the "suggested filter window" shown in that attached plot (although you need to pick its 2 widths yourself). Yeah, the resulting plot won't look as smooth as yours do, but on the one hand you complain about lack of high spatial frequency responce

if on the other hand you want plots as smooth as yours in the presence of the kind of noise you have... well good luck. Sorry, but you can't have it both ways.

[quote]

You're absolutely right. Using a low pass filter, it's not realistic to expect very smooth plots from such messy data.

Thanks!

d

 

Ok, I feel a little better now. barney was right that my NNA plot looked a little retarded. Let's try that one again.

Here's the NNA and savgol applied to previously mentioned ugly dyno data. The plot 0 is the NNA, plot 1 is the savgol:



Here's the same plot after applying more agressive smoothing:



Lo and behold, they are very similiar. This makes sense to me _now_, but again I have very limited experience with signal processing and zero knowledge of image filters.

d

 

I find it interesting that so much in depth mathematics is necessary to smooth a shape that the human eye can easily perceive; you could draw a line with your hand that would be nearly as accurate as some of the smoothing techniques. (Though a lot less repeatable.)

This got me into thinking. And, though I'm by no means highly qualified, it gave me some ideas. One thought is to use airmass/sec curves to guide the smoothing program as they are directly linked to power. But internal friction causes the peak airmass/sec to be slightly to the right of peak power, and ignition timing farks the whole deal up rather quickly in the likelihood of a knock-event. D'oh!

The other thing I noticed is that these rough curves look a lot like an audio-signal. The "noise" just looks like high-frequency vibrations in the dynamometer. I'd be curious to see if it were possible to simply turn down to the higher frequencies. The "smoothed" graphs just look like a low-frequency audio sample and the "unsmoothed" like a mixed low/high sample.

Since vibration is very likely the source of the erratic measurements in the first place, and since vibration usually happens at certain frequencies and harmonics, it seems natural to try to simply tune them out. Unfortunately I don't have the tools to even attempt to do this.

Just food for thought I suppose ...

Adrian~

 

Sometimes yes, sometimes no. There really isn't much mathematics in it at all (no proofs). A lot of processes that seem simple on the surface, turn out to me dazzlingly complex in practice.

One could argue that we're going about this is the wrong way and that what we should do is model generated kinetic energy and then try to fit the data. So, who wants to come up with the model?...Enjoy reading thousands of SAE back journals

They look sort of like an audio signal, but the number of samples is much, much lower. I tried to remove some of the sever outliers, but ended up introducing a scale bias that would throw off calibration attempts.

Actually, I blame the ugliness on the sampling rate. The event were're measuring isn't very long ( a few seconds) and we only get about 1000 samples total. I think it would clean up significantly with more samples and higher precision PC. That data isn't taken directly from the tach signal wire. That is a datalog from an Xede and the sampling is done by a script on my laptop.

d

 

Ahhh. I should have just said "computations". I've just managed the through Mult-var-calc and diff-eq and hadn't heard of some of these. But I've done nothing with signal processing/filtering.

Hah. Yeah, been there, done that. I actually have a pretty nice non-SAE doctoral thesis from Linkoping about ignition control modelling using an ion-sensor. It gives interesting pressure-modelling possibilities and the ion-sensor was the spark-plug, which makes it mechanically simple. With the pressure modelling you could get an extremely accurate pressure reading for the cyllinder. Integrating over the motion of the piston against the crank would give very detailed combustion analysis. But that's waaaaay overkill for these purposes. Anyway, if you want to read, here's a link to the PDF.

Ok, that makes more sense. I was sitting here wondering how on earth that much noise got picked up through the motion/flow sensors on whatever style dynamometer that was. It makes much more sense now. But, why not just run the RPM signal off the tach-wire? (Pardon my Evo n00bness if there's an obvious reason for that. My Viggen has Direct Ignition, but the tach wire is still easy enough to find.)

Sorry for butting into your discussion.

Adrian~

 

Yeah, you posted that to nasioc. It's on the stack of stuff for me to read when i get to my office tomorrow.

Oh, it's not problem at all if you want build a full service dyno system, but lots of people already do that. I'm interested only in processing logfiles generated by the Xede (or other ecu systems). The motivation is to up the sophistication of your at-home ecu mapper, not contend with mustang or dynojet.

Not a problem. Thoughtful input is always welcome.

d

 

I'm finding this discussion very interesting as well, but despite having taken many math courses, I have no idea what the hell anyone is talking about.

Anyone want to tutor me in math? )

 

The idea is that we have signal (an our case torque calculate by torque = dRPM/dTime), which is noisy. We want to filter out that noise so that patterns and features of the signal are visible to the eye. As it stands, raw input looks pretty much like a sea of points.

The typical method used in data analysis is called the linear regression. Specifically, most people use a least squares fitting (http://mathworld.wolfram.com/LeastSquaresFitting.html). This works very well if you have an model for the data that you're trying to fit. However, I've never seen a formulation of torque vs. RPM for a forced induction engine. We know general things about the shape of the curve, but I would need to find something more precise to use least squares. Thus I've been exploring other options.

I hesitate to call what I'm doing math, because it's very much of a black art. Even numerical recipes admits that this is a "murky area" where there is little theoretical foundation.

d

 

Hi Adrian, the problem with is that in the upper RPM range which is of major concern in high performance and racing applications is that the crankshaft is more ratcheting rather than rotating. Crankshafts are twisting and bending, rods are bending, twisting, and stretching, camshafts twisting and bending. The cam drive system also has some degree of distortion.


Kind regards
Shaun

 

*yawn*

Time to wake up a dead thread. I haven't looked at this problem in about 5 years, but now it's back.

I'm very curious to know what people are doing and especially how they are handling samples that are not uniformly distributed in time.

d

 

I do all my dataset analysis in Excel these days. I can accomplish most things using in-cell formulas, but if needed I use VBA. Using linear interpolation, should be pretty straightforward to convert a non-uniformly spaced dataset into a uniformly spaced dataset. I haven't looked at how I would implement it, but it seems that it might be possible to do it using an in-cell formula, and if not, I'm sure it can be done using VBA.

If you haven't already taken a peek, my code in VDR is just NNA. I never bothered with trying to make an evenly spaced dataset, but I think J_of_T added it to VDR to simplify the final data smoothing operation.

 

I'd be interested in seeing your spreadsheet. I'm amazed that JoT's vdr code produces a dyno plot that looks almost exactly like a recent dynojet pull of my car. My smoothing code produces similar results, but not feature-for-feature.

d