2U Supermicro servers are my go-to. These are much quieter than 1U servers, but the fans spin at 8800RPM. The IPMI fan modes available are Full (9000RPM), Heavy IO (6000RPM), and Optimal (supposed to auto-adjust). Unfortunately, setting the Fan Mode to Optimal seems to have a floor speed of 4500RPM, which is too loud. Even though my servers are in the garage, I sometimes work on projects there and can’t have it that noisy!
In the past, I’ve replaced Supermicro fans with low RPM Antec or Noctua fans, but since I moved my homelab to the garage I don’t need it to be that quiet.
Here’s my script, which tries to control the fan speed such that the noise is below that of a jet engine and the CPU temperature is below 50C. To avoid hunting, I set a minimum and maximum CPU range between 40-50C. The logic is simple:
- Set FAN speed mode to Full (this allows manual control of the fans).
- If the CPU is above 50C, bump the speed up by 10%
- If the CPU is below 40C, drop fan speed by 1%
- Repeat steps 2 and 3 every minute
⚠️ WARNING 1: Using this script could overheat and damage your CPU and other components; your server may release the magic smoke, burn through crystals like you’re traveling at Warp 10, or catch fire due to using this script. I just wrote it yesterday, so there may be bugs. I advise you not to use this script. If you go against my advice, keep a close eye on your temperatures.
⚠️ WARNING 2: This script was designed for a Supermicro X10 Motherboard. Yours may have a different fan configuration, in which case you will want to add or change the ipmitool commands.
# apt install python3
# apt install ipmitools
# vim /usr/local/bin/fan_control.py
#!/usr/bin/env python3
import os
import subprocess
import time
import syslog
import re
# Set your desired temperature range and minimum fan speed
MIN_TEMP = 40
MAX_TEMP = 50
MIN_FAN_SPEED = 5 # Sets an initial fan speed of 5%
current_fan_speed = MIN_FAN_SPEED
# IPMI tool command to set the fan control mode to manual (Full)
os.system("ipmitool raw 0x30 0x45 0x01 0x01")
time.sleep(2)
# Get the current CPU temperature
def get_cpu_temperature():
temp_output = subprocess.check_output("ipmitool sdr type temperature", shell=True).decode()
cpu_temp_lines = [line for line in temp_output.split("\n") if "CPU" in line and "degrees" in line]
if cpu_temp_lines:
cpu_temps = [int(re.search(r'\d+(?= degrees)', line).group()) for line in cpu_temp_lines if re.search(r'\d+(?= degrees)', line)]
avg_cpu_temp = sum(cpu_temps) // len(cpu_temps)
return avg_cpu_temp
else:
print("Failed to retrieve CPU temperature.")
return None
# Set the fan speed
def set_fan_speed(speed):
global current_fan_speed
current_fan_speed = speed
# Convert the speed percentage to a hex value
hex_speed = format(speed * 255 // 100, "02x")
# Set the fan speed for all 4 zones
os.system(f"ipmitool raw 0x30 0x70 0x66 0x01 0x00 0x{hex_speed}")
time.sleep(2)
os.system(f"ipmitool raw 0x30 0x70 0x66 0x01 0x01 0x{hex_speed}")
time.sleep(2)
# Log the fan speed change to syslog
syslog.syslog(syslog.LOG_INFO, f"Fan speed adjusted to {speed}%")
# Print the fan speed change to console
print(f"Fan speed adjusted to {speed}% - {hex_speed}")
# Set initial minimum fan speed
set_fan_speed(MIN_FAN_SPEED)
while True:
cpu_temp = get_cpu_temperature()
# Print the current CPU temperature to console
print(f"Current CPU temperature: {cpu_temp}°C")
if cpu_temp > MAX_TEMP and current_fan_speed < 100:
# Increase the fan speed by 10% to cool down the CPU
new_fan_speed = min(current_fan_speed + 10, 100)
set_fan_speed(new_fan_speed)
elif cpu_temp < MIN_TEMP and current_fan_speed > MIN_FAN_SPEED:
# Decrease the fan speed by 1% if the temperature is below the minimum threshold
new_fan_speed = max(current_fan_speed - 1, MIN_FAN_SPEED)
set_fan_speed(new_fan_speed)
# Wait for 60 seconds before checking the temperature again
time.sleep(60)
# vim /etc/systemd/system/fan_control.service :
[Unit]
Description=Fan Controller Service
After=network.target
[Service]
Type=simple
User=root
ExecStart=/usr/bin/python3 /usr/local/bin/fan_control.py
Restart=on-failure
[Install]
WantedBy=multi-user.target
Set file permissions:
# chmod 755 /usr/local/bin/fan_control.py
# systemctl daemon-reload
# systemctl enable fan_control.service
# systemctl start fan_control.service
# systemctl status fan_control.service
Bugs: so, I’ve come across one bug where the script won’t spin down both fan zones, but killing the script and running it a second or third time seems to work. I suspect this is from sending ipmi commands too fast, so I’ve added sleep delays, but because it’s an intermittent issue who knows if that fixed it.
Improvement ideas: It would be great to add temperature monitoring for the other components (HDD, etc.). But in my observation, if I can keep the CPU cool the rest of the components are okay.
To watch the status, check IPMI or tail syslog:
# tail -F /var/log/syslog |grep -E ‘speed|temp’
Fan speed adjusted to 4% - 0a
Current CPU temperature: 38°C
Fan speed adjusted to 3% - 07
Current CPU temperature: 39°C
Fan speed adjusted to 2% - 05
Current CPU temperature: 39°C
Fan speed adjusted to 1% - 02
...
Current CPU temperature: 48°C
Current CPU temperature: 50°C
Current CPU temperature: 52°C
Fan speed adjusted to 11%
Current CPU temperature: 51°C
Current CPU temperature: 48°C
Current CPU temperature: 46°CThat was set up last night.
When I woke up this morning, the fans were running at a pleasant hum of 1800 RPM.
He who blesses his friend with a loud voice early in the morning, It will be counted as a curse to him. – Proverbs 27:14 LSB
I re-did this with a Windows powershell slant. This is known working on X11 Motherboards but should work on 9 and 10 as well (and likely 12)
feel free to publish as an adder to what you did.
# Function to get CPU temperature using IPMI
function GetCpuTemperatureFromIPMI {
$ipmiOutput = & “ipmicfg-win.exe” -sdr
$cpuTemp = $ipmiOutput | Where-Object { $_ -match ‘\(4\) CPU Temp\s+\|\s+(\d+(?:\.\d+)?)C/’ } | ForEach-Object { $matches[1] }
return [double]$cpuTemp
}
# Function to set fan mode to auto using IPMICFG-Win.exe
function SetFanModeAuto() {
Start-Process -FilePath “IPMICFG-Win.exe” -ArgumentList “-fan 0” -Wait
}
# Function to set fan mode to quiet using IPMICFG-Win.exe
function SetFanModeQuiet() {
Start-Process -FilePath “IPMICFG-Win.exe” -ArgumentList “-fan 1” -Wait
Start-Process -FilePath “IPMICFG-Win.exe” -ArgumentList “-raw 0x30 0x70 0x66 0x01 0x00 0x06” -Wait
}
# Main script logic
try {
$currentFanMode = $null # Variable to store the current fan mode
while ($true) {
$cpuTemperature = GetCpuTemperatureFromIPMI
if ($currentFanMode -ne 1 -and $cpuTemperature -lt 50) {
SetFanModeQuiet
$currentFanMode = 1 # Set the current fan mode to quiet (1)
Write-Host “CPU temperature is below 50°C. Fan mode set to quiet.”
} elseif ($currentFanMode -ne 0 -and $cpuTemperature -ge 53) {
SetFanModeAuto
$currentFanMode = 0 # Set the current fan mode to auto (0)
Write-Host “CPU temperature is above 53°C. Fan mode set to auto.”
}
# Wait for 20 seconds before the next check
Start-Sleep -Seconds 20
}
} catch {
Write-Host “Error occurred: $_”
}
Thanks for sharing, Jeff!
Have you thought of putting your Script on Github or something like that ?
I think I’ll have to adapt it quite a bit, including reading drive Temps via Smartctl and HBA temps via LSIutil, then put everything on Github (and give you the Credit you deserve for it !).
My main problem isn’t the CPU, it’s the Hard Drives and the LSI HBA. I replaced the 80mm Fans in my 2u Chassis with some quieter ones (Arctic F8 PWM that barely spin at 1800 rpm, provide around 30 CFM at ONLY 1 mmH2O Static Pressure), similarly to what you did in your other Article.
I never had a Problem with a Setup like this in the Past, but maybe the combination of higher Power Loss of each Drive (compared to previous Systems) and the fact that there are 12 HDDs installed instead of 8 HDDs made all the Difference. After 2 Hours of running SMART Test, HDDs reached 90 degrees Celsius for a while (under 2 Hours, probably [much] less than that, as the whole test took 2 Hours). HBA went above 130 Degrees Celsius …
Hi, Stefano. I have not uploaded to GitHub because I don’t have time (I’ve put a few things on GH in the past and it ended up taking too much of my time servicing PRs); but you’re welcome to put it on GitHub or distribute it wherever. If you need a license consider it MIT licensed.
Hi Benjamin. I can imagine that sometimes happens when you are popular ;). I never get any PR and when I fork a Project and submit a PR it gets totally ignored. At the end of the day I more or less use Github to store the Code for my Projects, unless it’s something ultra Private or something like that, which is very rarely the case anyways. In terms of License I usually go with AGPL License (GPL might be sufficient for most cases, but I though the “A” for hosted environments might also be good). A quick Google Search indicates that integrating MIT Code into AGPL Project seems fine (NOT the other way around though). Or would you be fine with AGPL as well ? Thanks. Kind Regards, Stefano.
Yes. AGPL is fine. I believe you’re right that you fork an MIT project as GPL or AGPL anyway, but I’m not a lawyer so don’t quote me on that! Thanks, Stefano.
Alright. I started developing it a bit further. So far I included the Drives (HDD/SSD/NVME) Temperature Control in the Algorithm and config done via YAML File.
For those Interested:
https://github.com/luckylinux/supermicro-fan-control
I think I’ll also have to do a “Fail-Safe” Systemd Service in pure BASH to trigger an emergency shutdown in case the Python Library has some glitch …
Nice work Stefano, I like the additional temperature sensors, incidentally I just had a drive failure and wonder if the HDDs may be running too hot!
That’s sad to hear Benjamin (about the failed drive).
Unfortunately I had to do this after several of my HDDs (bought second hand thankfully) went above 80°C … Yeah, Celsius. 1 Drive later failed a Badblocks Test and 11 other Drives later somewhat survived and passed a Badblocks test.
I just changed the code slighly to use subprocess instead of os.system in most parts of the Code, as somebody else running my Code didn’t notice any errors yet the fans were kept spinning like before.
The logging Feature will probably need some work too. Right now I’m logging EVERYTHING since I want to check what it’s doing at all times.
Anyways, I suggest you also install https://github.com/luckylinux/cooling-failure-protection as a last-resort (the feature is now integrated into supermicro-fan-control, but I think it’s always better to have somewhat of a redundancy e.g. Python Script fails, Config File has errors and breaks everything, etc). That’s a BASH Fallback Basically.
That will BEEP (warning) and later shutdown the Server in case the DRIVES (no CPU protection right now !) are getting very high due to e.g. Fans not having sufficient airflow/static pressure or simply fan failure.
Actually, I had another Finding.
Not sure if all Motherboards are affected but 0xFF is actually NOT the maximum speed. In some cases it’s 0x64. See here for instance: https://forums.servethehome.com/index.php?threads/supermicro-x9-x10-x11-fan-speed-control.10059/page-10
If I went above 0x64 all kind of strange things would start to occur. The System would work, at full Speed (without your code “Version” at least), but for all the wrong Reasons :D.
That’s why your * 255 // 100 was a bit confusing for me. I trusted it at first, but (especially with low-power Fans), they would always go full-blast. And I couldn’t figure out why !
If you go “too high”, I believe an Overflow occurs, Fan Speed drops to 0 rpm (Non-Recoverable Error), then the Supermicro IPMI/BMC Controller kick the Fan to Full Speed.
In your Version, you don’t set the Fan Speed if it did NOT change. So the Fans will be stuck at Full Speed (which is good in a way).
In my Code on the other Hand, I decided to do a “set” every time. I hope it will NOT wear out the Flash Memory of the IPMI Device. I hope all of these “set” go into RAM, not Flash ! I did that to avoid that we play with ipmitool outside of the Program, then forget to restart the Service.
I updated my Code and implemented all of these Changes. There is now a per-zone Parameter in the Config File (by Default set to 0x64, not 0xFF like we first thought) for the Maximum Speed Value supported by the IPMI Device.
Fixed in -> https://github.com/luckylinux/supermicro-fan-control/commit/4d1f733b134bb3965125273366987f5165a9330f
Some testing I did: https://github.com/luckylinux/supermicro-fan-control/issues/1#issuecomment-2208320647
Would be great if you could give it a try. Because I fear that, if you need to go above say 40%, you’d suffer from the same “Overflow” issue (internal to the BMC/IPMI Chip I mean).
Or try to run ipmitool with 0xff as your last input. Check noise and IPMI Event Log. Chances are that fan speed dropped to almost zero, then Supermicro IPMI Controller forced them at full Speed.
Only way to know is to look at the IPMI Event Log. Now I am also spamming my Log and write to it all IPMI Events ;).
And of course I released another update to fix an Error if the System Event Log is empty. Sorry for the Duplicate Comment :(.
This is why I always wait a few months for things to settle before trying anything. |:-)
This seems to work partially with a X9 board also, but there’s only two fan speeds – slow and fast.
ipmitool -I lanplus -H (IP of IPMI) -U (Username) -P (Password)
raw 0x30 0x45 1 3 (slow)
ipmitool -I lanplus -H (IP of IPMI) -U (Username) -P (Password)
raw 0x30 0x45 1 1 (fast)
Is there’s a way to incorporate this into your script? As an example, start with the slow speed, and if the CPU temp rises above 50C, adjust to fast and then adjust back to slow?
I’m not talented enough in python to do it myself unfortunately.
The control principle would be similar except that it’s just histeresis and no proportionality/variable range . You are also using ipmi tool via IP address and not local to the server, so it’s different in that regards anyway.
I tried to do my script as generic as possible but not quite yet able to support these 2 new things.
Are you sure you only want to look at CPU temperature though? That might be dangerous (especially for ssd/hdd).
To implement is the easy part. The hard part is if you considered the consequences of it (besides fans ramping up/down all the time).