?

Log in

No account? Create an account

Previous Entry | Next Entry

Dear lazyweb: CPUFreq ondemand = fail

I have managed to stay away from video games for the past few months but decided to pick up one of my old favorites and spend a little time playing. Much to my dismay, 3D performance had regressed since the last time I played a full-on video game. Most FOSS and commercial games were running at 10 FPS or so. 32 and 64 bit. 100% CPU utilization.

Well, after running some oprofile and sysprof stuff, I discovered that some 60% of the CPU was being spent on in-Xorg and in-Kernel video computation. So, I thought surely it was related to some new DRI2 stuff I was running. So... I went backwards two years in Intel graphics driver development. No change; little faster. I went forward to the very latest and greatest stuff (and X, kernel and userspace Mesa): Gallium3D with DRI2 state tracker on KMS. 30% speed up but still ~15 FPS on average. So it wasn't the driver...

Finally, I decided to pull out the big guns and SSH in to my laptop while playing to watch various things. During this, I discovered that CPUFreq ondemand never raised my frequency above 800MHz. "What?" I thought, "Surely ondemand is still reliable... isn't it?" We've been recommending to people that they always use ondemand since at least the 2.6.9 days. So, after:

sudo cpufreq-set -c 0 -g performance
sudo cpufreq-set -c 1 -g performance

And running every game I have, beautifully, at generally around 60 FPS (vsync), I was armed with new information to Google. And indeed, there are reports starting around mid-last year that ondemand was having major regressions. And they appear to have been largely ignored, so far.

So, does anyone know what the hell is going on with ondemand?

Here's some anecdotal evidence which may help: computational benchmarks which do 100% of their calculation in user-space correctly scale the CPU; anything which performs a portion of its work inside the kernel (graphics related stuff) does not scale the CPU

For now, my laptop has roughly 50% battery life. *sigh*

Update 2009-04-12: I had been building my own kernel throughout the mentioned series and then switched to the official Debian 2.6.29 series--the problem was still there. A week ago, I switched to Fedora--to get closer to Gnome development in more aspects--and discovered that Fedora's build of 2.6.29 is not affected. I haven't had a chance to look it what the underlying differences might be but perhaps some enterprising person will beat me to it.

Comments

( 33 comments — Leave a comment )
(Anonymous)
Mar. 23rd, 2009 04:50 am (UTC)
ondemand fail =(
I've been using my old gateway laptop for gaming for about 2 or 3 years and i ALWAYS noticed this ondemand problem. Generally, it even bugs and i am forced to disconnect the power from the laptop and reconnect again so the processor can scale correctly again, very noisy :(
ext_176827
Mar. 23rd, 2009 06:11 am (UTC)
Re: ondemand fail =(
Hm... I wonder if this has anything to do with the crackly audio issues people with Ubuntu Jaunty have been having?
Good thing there's a cpu frequency applet for the occasion :)
(Anonymous)
Mar. 23rd, 2009 07:39 am (UTC)
You might want to use conservative instead of ondemand. The ondemand scheduler jumps up and down from minimal to maximum frequency all the time.
jasondclinton
Mar. 23rd, 2009 02:58 pm (UTC)
The point is that ondemand used to just work and it is the default *everywhere* because it's supposed to *always* just work.
(Anonymous)
Mar. 23rd, 2009 10:30 am (UTC)
sleeping
As far as I remember, with modern CPUs, putting the CPU to full-speed and making it sleep as often as possible is more energy efficient than scaling the speed.

Obviously though games will use any CPU time that's available, so without throttling you'll be left with more speed and more energy usage.

Be sure to test the actual battery life with both settings though.
jasondclinton
Mar. 23rd, 2009 03:01 pm (UTC)
Re: sleeping
Yes, that is correct. But that doesn't address why it has stopped scaling correctly. Gaming used to work just fine with ondemand. Now it fails.
Re: sleeping - jasondclinton - Mar. 23rd, 2009 03:02 pm (UTC) - Expand
(Anonymous)
Mar. 23rd, 2009 11:02 am (UTC)
Did you try powertop? (http://www.lesswatts.org/projects/powertop/)

It is very likely that ondemand does switch up and back again.
When playing a game here It is around 60% in highest speed and rest in lowest.
jasondclinton
Mar. 23rd, 2009 03:06 pm (UTC)
Yes, I tried powertop over the SSH connection. That was the easiest way to quantify now much time was spent at 800MHz instead of 2.4GHz. While gaming: 98%.
(Anonymous)
Mar. 23rd, 2009 11:25 am (UTC)
Change default threshold!
By default ondemand scales the CPU only when its usage is more than 80%.
To change the value, use this command (for example for 50%).

echo 50 > /sys/devices/system/cpu/cpu0/cpufreq/ondemand/up_threshold
jasondclinton
Mar. 23rd, 2009 03:07 pm (UTC)
Re: Change default threshold!
Usage is 100% at 800MHz and it still doesn't scale. I guess I didn't make it clear that the game is CPU-bound.
Re: Change default threshold! - (Anonymous) - Apr. 13th, 2009 04:01 pm (UTC) - Expand
(Anonymous)
Mar. 23rd, 2009 11:28 am (UTC)
Do not ignore nice load
Another hint I forgot could be the nice load calculation
echo 0 > /sys/devices/system/cpu/cpu0/cpufreq/ondemand/ignore_nice_load
jasondclinton
Mar. 23rd, 2009 03:08 pm (UTC)
Re: Do not ignore nice load
Xorg and the kernel DRI bits are not 'niced' so this is irrelevant. The default for this is sane. (Unless the implementation has a bug confusing signedness; I'll test it just to be sure.)
(Anonymous)
Mar. 23rd, 2009 02:31 pm (UTC)
Me Too
I just discovered this a few weeks ago myself.

With my Fedora 10 laptop I've installed newer driver stuff from rawhide to get it up to DRI2 status and get stable Intel performance.

I noticed that I get a 30-50% boost in performance if I disable ondemand.

To examples that I can think of off the top of my head is:

Maniadrive, car racing game. Disabling ondemand took the fps from the 30-60 range to the 70-90 range.

for playing large Adobe Flash videos, like the stuff you can get from Hulu.com, (which unfortunately won't work with gnash or anything else yet) I have OpenGL forced on. You can do this with:
OverrideGPUValidation=true

in:
/etc/adobe/mms.cfg

This gets you good flash performance under Intel UXA stuff in DRI2-land. Even works well with Compiz.

While with ondemand I may get choppy video playback on moderately busy parts of movies. With it disabled I get much smoother results.

---------------------


And ya this sort of thing can cause stuttering in Pulse-audio. Any time your going to have a video go out of sync due to bad video performance or anything like that it is going to cause cascading problems.

However: A performance tip for pulseaudio is to:
A. give yourself the ability to run with realtime priorities (edit /etc/security/limits.conf, add yourself to pulse-rt or whatever else is appropriate for your system.)

This will solve a lot of stuttering issues, but will not help if PA is consuming lots of CPU time.

So...

B: Change the mixer resampler method. In /etc/pulse/daemon.conf you can find something like:

; resample-method = speex-float-3

This is moderately high quality sampling method, but unless your using something like a nice Core Duo cpu or better it is going to eat a irritatingly large amount of CPU time.

I find that with
resample-method = ffmpeg

I get good results on my 1.6ghz Atom-based Netbook, which has about the cpu performance of a 600mhz-1ghz Pentium3.

With resample-method = trivial you can get about the same level of overhead that you get from simply using dmix.

(I think. I can't find anywere that documents the method used by dmix)

So people with older machines should look into that. It is then quite a bit better then simply pages and pages of Ubuntu forums telling people to shut off pulse audio because it is shit.


And the same thing with DRI2 and ondemand...
(Anonymous)
Mar. 23rd, 2009 02:40 pm (UTC)
I don't know what the deal with the ondemand governor is but ever since I moved to an AMD CPU it is actually annoyingly 'laggy' with detecting the load on my CPU. With my Intel CPU it would switch up instantly and wasn't even noticable, with my AMD CPU I'm waiting up to a second for it to increase speed (like when scrolling a webpage or something) even if I tune the sampling_rate_min/max and up_treshold parameters under /sys/devices/system/cpu/*/cpufreq/ondemand/. Maybe it is just a limitation of AMD CPUs though.. I just don't remember it being laggy like this with my old Athlon XP (current cpu is Athlon64 X2)

I think I should just set it up so that it runs with the 'performance' governor unless my screensaver is active..
jasondclinton
Mar. 23rd, 2009 03:09 pm (UTC)
Trying to set policy like this is a bad idea. See Mathew Garret's posts about trying to outsmart the kernel on power. The right solution is to fix ondemand.
(no subject) - (Anonymous) - Mar. 23rd, 2009 03:29 pm (UTC) - Expand
(no subject) - jasondclinton - Mar. 23rd, 2009 03:34 pm (UTC) - Expand
ext_176913
Mar. 23rd, 2009 05:11 pm (UTC)
cpu scaling nightmare
I've been having problems with cpu scaling for at least a year now, iirc. It goes on and off and it's really hard to pinpoint - depends on whether I remove power during suspend or if I reboot it fully, sometimes it works, sometimes it doesn't... you're lucky that you can actually make the cpu go full speed, a lot of the times it gets stuck on the lowest setting no matter what. Sometimes the only way to make it go when running from power is to remove and connect the power again (!!)

When on power, the scaling misbehaves dramatically, and cpufreq-info has actually told me it can only go from 800mhz to 800mhz. The cpufreq* scripts have, for some reason, lost their suid bit, which means the applet can't use them to set anything, and from the behaviour after I restored them, it looks like the system itself doesn't do scaling properly without it. The ondemand governor, that should jump from 800 to 2.0, is scaling like the conservative governor instead for some reason. Having anything except the performance governor = always at 800 for most things (including FF, any IDEs, compiling, ...). I personally know people who bought laptops and have them completely crippled 99% of the time with no idea how to fix it... it's not only the ondemand governor that's broken, there's real problems going on here with all the scaling pieces. :(
(Anonymous)
Mar. 23rd, 2009 07:46 pm (UTC)
Does any kernel do this correctly?
If so have you been able to narrow this down using git-bisect?
jasondclinton
Mar. 23rd, 2009 08:37 pm (UTC)
Re: Does any kernel do this correctly?
No, I haven't git bisected yet. It's extremely time consuming and I've already sunk around 10 hours in to this. Plus, which module is to blame? ACPI, cpufreq, nohz, unified arch?

Additionally, subsequent to my posting of this, I have gotten a number of "me too's" from very smart people whom have had no luck tracking this down. So, I'm not optimistic.
Re: Does any kernel do this correctly? - (Anonymous) - Mar. 23rd, 2009 11:32 pm (UTC) - Expand
Re: Does any kernel do this correctly? - (Anonymous) - Mar. 25th, 2009 11:22 am (UTC) - Expand
ext_96809
Mar. 24th, 2009 12:13 am (UTC)
eh?
Wait, what?

I've always played with the scheduler set to ondemand, and can't complain about any perf issues (played quake wars on highest settings, currently on savage2 with medium-high). 8600m gt here.

I'll try 'performance' tonight though.
jasondclinton
Mar. 24th, 2009 12:15 am (UTC)
Re: eh?
If you're using a kernel that's older than last summer, then we probably don't need to know.
Re: eh? - ext_96809 - Mar. 24th, 2009 03:27 pm (UTC) - Expand
ext_96809
Mar. 31st, 2009 12:09 am (UTC)
No, disappointingly enough, I get no massive performance spike when setting it to performance. Same as ondemand.

8600m gt, ubuntu 8.10, nvidia drivers.
jasondclinton
Mar. 31st, 2009 01:35 am (UTC)
Yea, kernel old enough to not be affected.
Which distro do you use? - (Anonymous) - Apr. 1st, 2009 03:49 pm (UTC) - Expand
Re: Which distro do you use? - jasondclinton - Apr. 1st, 2009 03:50 pm (UTC) - Expand
ext_183868
Apr. 27th, 2009 10:52 pm (UTC)
Have you filed a bug on launchpad.net
I've filed a bug on launchpad.net with similar comments. See https://bugs.launchpad.net/ubuntu/+source/linux/+bug/367739 and https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/368231 , both of which are related to this issue, I believe.

I'm really disturbed to think of how many 9.04 users might be running with a 50% clock speed and never know it.
ext_203491
Aug. 13th, 2009 05:05 pm (UTC)
Change in Linux 2.6.30
I am a Gentoo user and may have run into a similar problem (ondemand worked fine on Linux <=2.6.29 versions then ceased working properly with 2.6.30). I finally tracked this down to an explicit change to p4-clockmod.c (.../arch/x86/kernel/cpu/cpufreq/p4-clockmod.c) where the CPU transition latency was changed from 1000000 to 10000001 with a comment that this is to prevent the ondemand governor from changing the CPU speed.

Log messages are generated (in kern.log or similar log files) when one attempts to switch from performance governor to ondemand governor, to the effect of:
"ondemand governor failed, too long transition latency of HW, fallback to performance governor"
which may indicate that this "feature" may be "active". I have not verified whether or not the source change(s) are the same as those involved in some other comments.

The kernel comments suggest that this was done due to long latencies in speeding up or slowing down the CPU which I find to not be the case with my Pentium IV Prescott (P4D). CPU scales down or up relatively easily (within a few seconds or less) based on load. This has a real impact on power use (the machine (a significantly upgraded HP Pavilion a630n) drops from ~147W @ 2.8 GHz to ~107W @ 350 MHz = ~25% savings). I think there may be a push by kernel developers to encourage people to move to using the more complex cpufreqd approach to power conservation which may be a bit of overkill for typical desktop (AC-always-on) systems.

It is also worth noting that the Linux configuration "help" regarding X86_P4_CLOCKMOD is a bit misleading. I do not find that the "slowdowns"/"latencies" it allows are "excessive" (indeed they seem fine for normal desktop use). It also seems that having P4_CLOCKMOD in the kernel may be required for other power utilities (Gnome Frequency Scaling Monitor for example) to function because it seems to be required to enable the creation of the "/sys/devices/system/cpu/cpu0/cpufreq/*" files which those utilities use.

Robert Bradbury (robert.bradbury@gmail.com)
( 33 comments — Leave a comment )

Profile

color, uphair, smile
jasondclinton
Jason D. Clinton

Latest Month

September 2011
S M T W T F S
    123
45678910
11121314151617
18192021222324
252627282930 

Tags

Powered by LiveJournal.com
Designed by Tiffany Chow