There are numerous small molecular compounds around us to affect our health, such as drugs, pesticides, food additives, industrial chemicals, and environmental pollutants. Over decades, properties related to absorption, distribution, metabolism, excretion, and toxicity (ADMET) have become one of the most important issues to assess the effects or risks of these compounds on human body. Recent high-rate drug withdrawals increase the pressure on regulators and pharmaceutical industry to improve preclinical safety testing. Since in vivo and in vitro evaluations are costly and laborious, in silico techniques have been widely used to estimate these properties. In this review, we would briefly describe the recent advances of in silico ADMET prediction, with emphasis on substructure pattern recognition method that we developed recently. Challenges and limitations in the area of in silico ADMET prediction were further discussed, such as application domain of models, models validation techniques, and global versus local models. At last, several new promising research directions were provided, such as computational systems toxicology (toxicogenomics), data-integration and meta-decision making systems, which could be used for systemic in silico ADMET prediction in drug discovery and hazard risk assessment.