Abstract: In the actual drilling process, the effects of cutting formation drill, drill string and borehole face collisions will certainly create a drilling tool vibration. These vibrations in inclinometer acceleration sensor output are reflected, so that acceleration sensor output value can not be true reflect borehole posture changes. In order to obtain accurate borehole attitude, it is necessary to study the sensor output during the vibration condition, so as to eliminate the effects of vibration. This paper begins with the raw data of the indoor simulation vibrational experiments, gets the general relationship between vibrational frequency and rotational speed through the power spectral density estimation of sensor output signal which is based on Matlab, and designs the specific filter. Through analyzing and processing the experimental data, energy of vibration interference signal is greatly attenuated, output fluctuation of three sensitive axis inclinometer sensor is significantly reduced, so the validation of the method is feasible in rotary steering system.