Empirical review of automated analysis tools on 47,587 Ethereum smart contracts

Over the last few years, there has been substantial research on automated\nanalysis, testing, and debugging of Ethereum smart contracts. However, it is\nnot trivial to compare and reproduce that research. To address this, we present\nan empirical evaluation of 9 state-of-the-art automated analysis tools using\ntwo new datasets: i) a dataset of 69 annotated vulnerable smart contracts that\ncan be used to evaluate the precision of analysis tools; and ii) a dataset with\nall the smart contracts in the Ethereum Blockchain that have Solidity source\ncode available on Etherscan (a total of 47,518 contracts). The datasets are\npart of SmartBugs, a new extendable execution framework that we created to\nfacilitate the integration and comparison between multiple analysis tools and\nthe analysis of Ethereum smart contracts. We used SmartBugs to execute the 9\nautomated analysis tools on the two datasets. In total, we ran 428,337 analyses\nthat took approximately 564 days and 3 hours, being the largest experimental\nsetup to date both in the number of tools and in execution time. We found that\nonly 42% of the vulnerabilities from our annotated dataset are detected by all\nthe tools, with the tool Mythril having the higher accuracy (27%). When\nconsidering the largest dataset, we observed that 97% of contracts are tagged\nas vulnerable, thus suggesting a considerable number of false positives.\nIndeed, only a small number of vulnerabilities (and of only two categories)\nwere detected simultaneously by four or more tools.\n