Macroprudential stress tests have been employed by regulators in the United States and Europe to assess and address the solvency condition of financial firms in adverse macroeconomic scenarios. We provide a test of these stress tests by comparing their risk assessments and outcomes to those from a simple methodology that relies on publicly available market data and forecasts the capital shortfall of financial firms in severe market-wide downturns. We find that: (i) The losses projected on financial firm balance-sheets compare well between actual stress tests and the market-data based assessments, and both relate well to actual realized losses in case of future stress to the economy; (ii) In striking contrast, the required capitalization of financial firms in stress tests is found to be rather low, and inadequate ex post, compared to that implied by market data; (iii) This discrepancy arises due to the reliance on regulatory risk weights in determining required levels of capital once stress-test losses are taken into account. In particular, the continued reliance on regulatory risk weights in stress tests appears to have left financial sectors under-capitalized, especially during the European sovereign debt crisis, and likely also provided perverse incentives to build up exposures to low risk-weight assets.