Ira Chinoy /
DATA ON THE WEB
A sample of sources
This site has examples of data that can
be downloaded from the Web or searched on the Web
Lists of links to data sources on the web:
Ø Downloadable Data on the Net (NICAR ‘Net Tour’): http://www.ire.org/training/nettour/papertrails.html
Ø A Journalist’s Database of Databases (Drew Sullivan): http://www.drewsullivan.com/database.html
Ø Finding Data on the Internet (RobertNiles.com): http://nilesonline.com/data/
Ø FedStats – Government site with links to over 100 federal agencies with data on the web: http://www.fedstats.gov/
Ø GPO Access (Government Printing Office): Search multiple federal databases from one page: http://www.gpoaccess.gov/multidb.html
Governments with sites for downloadable data:
Federal government: Data.gov: http://www.data.gov/
District of Columbia: Data Catalog: http://data.dc.gov/
Governments with sites for summary data:
Maryland: StateStat: http://www.gov.state.md.us/statestat/
Baltimore: CitiStat: http://www.baltimorecity.gov/Government/AgenciesDepartments/CitiStat.aspx
Business
Ø Maryland Department of Assessment and Taxation:
o List of searchable databases: http://www.dat.state.md.us/
o Business Data Search: http://sdatcert3.resiusa.org/ucc-charter/
o Real Property (real estate) Data Search: http://sdatcert3.resiusa.org/rp_rewrite/
Culture and entertainment
Ø Cultural Policy & The Arts National Data Archive: http://www.cpanda.org/
Ø UNESCO Institute for Statistics: http://www.uis.unesco.org/ev.php?URL_ID=5275&URL_DO=DO_TOPIC&URL_SECTION=201
Ø Federal Communications Commission – Registered cable communities: http://www.fcc.gov/mb/engineering/liststate.html
Demographics and social issues, current and historical:
Ø Census:
o 2000 Census Data for
o Historical statistics from census data: http://www.census.gov/statab/www/minihs.html
Ø Kids Count – Annie E. Casey Foundation compilation of data on the well-being of children: http://www.aecf.org/kidscount/
o Overview: http://www.aecf.org/kidscount/databook/
o Downloadable raw data: http://www.aecf.org/cgi-bin/kc.cgi?action=newdata
o Kids Count Census data online: http://www.aecf.org/cgi-bin/aeccensus.cgi?action=dataresults&area=24S
Ø The Association of Public Data Users: http://www.apdu.org/
Ø Inter-university Consortium for Political and Social Research: http://www.icpsr.umich.edu/access/index.html
Ø The Urban Institute:
o State Database: http://www.urban.org/Content/Research/NewFederalism/Data/StateDatabase/StateDatabase.htm
o National Survey of America’s Families: http://www.urban.org/Content/Research/NewFederalism/NSAF/Overview/NSAFOverview.htm
Ø
Economy, business, financial institutions, nonprofits, workplace:
Ø Internal Revenue Service – Tax Stats – Nonprofits, by state: http://www.irs.gov/taxstats/charitablestats/article/0,,id=97186,00.html
Ø
o National Compensation Survey Tables: http://www.bls.gov/ncs/home.htm#tables
o Worker Fatalities Data: http://www.bls.gov/iif/oshcfoi1.htm
Ø
Ø National Credit Union Administration call reports: http://www.ncua.gov/data/FOIA/foia.html
Ø Federal Deposit Insurance Corporation - custom data downloads: http://www2.fdic.gov/sdi/main.asp
Ø
o Searchable database of workplace injuries: http://www.osha.gov/pls/imis/establishment.html
Ø
o Searchable databases: http://sdatcert3.resiusa.org/ucc-charter/CharterSearch_f.asp
Education:
Ø College Results Online – data on colleges and universities compiled by The Education Trust: http://www.collegeresults.org/
Ø
Ø Education Week – student achievement: http://www.edweek.org/sreports/qc02/reports/achieve-t1.htm
Ø
o Common Core of Data – Education Data Files: http://nces.ed.gov/ccd/ccddata.asp
o Demographics of school districts: http://nces.ed.gov/surveys/sdds/downloadmain.asp
o Digest of Education Statistics, 2002: http://nces.ed.gov/programs/digest/d02/list_tables.asp
Ø U.S. Department of Education, Federal Student Aid – school default rates: http://www.ed.gov/offices/OSFAP/defaultmanagement/cdr.html
Ø UNESCO Institute for Statistics: http://www.uis.unesco.org/ev.php?URL_ID=5275&URL_DO=DO_TOPIC&URL_SECTION=201
Environment, weather, natural disasters and other emergencies:
Ø
Historical
Severe Weather Data – NOAA: http://www.spc.noaa.gov/archive/
Ø Right-to-Know Network – Environmental Databases: http://www.rtk.net/rtkdata.html
Ø Toxicology Data Network TOXNET: http://toxnet.nlm.nih.gov/
Ø
o EPA Data: http://www.epa.gov/epahome/Data.html
o EPA Envirofacts: http://www.epa.gov/enviro/
o EPA Drinking water data: http://www.epa.gov/safewater/data/getdata.html
o Engangered Species Protection Program Databases: http://www.epa.gov/espp/database.htm
Ø
o Home page: http://www.nrc.uscg.mil/nrchp.html
o Downloadable data (in Excel): http://www.nrc.uscg.mil/download.html
o Searchable data: http://www.nrc.uscg.mil/foia.html (slow)
Government spending:
Ø Office of Management and Budget: searchable databases of contracts and grants -- http://www.fedspending.org/
Ø Census: Federal, State, and Local Government Data: http://www.census.gov/govs/www/
Ø Consolidated Federal Funds Report: http://www.census.gov/govs/www/cffr.html
Ø Federal Assistance Award Data System: http://www.census.gov/govs/www/faads.html
Ø D.C. Office of Contracting and Procurement (OCP) – searchable database: http://app.ocp.dc.gov/RUI/information/awards/all_awards.asp?radio=3
Health:
Ø American Public Human Services Association – Federal Data Index: http://www.aphsa.org/links/federaldata.asp
Ø Kaiser Family Foundation / State Health Facts Online: http://www.statehealthfacts.kff.org/cgi-bin/healthfacts.cgi?action=rawdata
Ø Excluded
Individual and Entities, Office of the Inspector General,
o http://oig.hhs.gov/fraud/exclusions/listofexcluded.html
Ø Medicare databases: http://www.medicare.gov/Download/DownloadDB.asp [Note: site was inaccessible 5-23-04]
o comparing nursing homes (including inspections): http://www.medicare.gov/NHCompare/Static/Related/DownloadDB.asp?dest=NAV|Home|Resources|DownloadDatabase#TabTop
· Notes about the data: http://www.medicare.gov/NHCompare/Static/Related/DataCollection.asp?dest=NAV|Home|DataDetails|DataCollection#TabTop
o home health care: http://www.medicare.gov/hHCompare/Static/Related/DownloadDB.asp?dest=NAV|Home|Resources|DownloadDatabase#TabTop
o dialysis facilities: http://www.medicare.gov/Dialysis/Home.asp
Ø Centers for Medicare and Medicaid Services: http://www.cms.hhs.gov/researchers/
Ø U.S. Health Resources and Services Administration: Geospatial Data Warehouse: http://datawarehouse.hrsa.gov/default.htm
Ø Manufacturer and User Facility Device Experience Database - (MAUDE): http://www.fda.gov/cdrh/maude.html
Ø Centers for Disease Control and Prevention:
o CDC Data and Statistics: http://www.cdc.gov/node.do/id/0900f3ec8000ec28
o
Ø
Ø National Practitioner Databank: http://www.npdb-hipdb.com and http://www.npdb-hipdb.com/publicdata.html
Ø
o Food Stamp Program: http://www.fns.usda.gov/pd/fspmain.htm
o Child Nutrition: http://www.fns.usda.gov/pd/cnpmain.htm
o WIC (Women, Infants and Children): http://www.fns.usda.gov/pd/wichome.htm
o Food Distribution Program: http://www.fns.usda.gov/pd/fdpmain.htm
Housing:
Ø
o
Data sets: http://www.huduser.org/datasets/pdrdatas.html
o
Subsidized
housing: http://www.huduser.org/datasets/assthsg/statedata98/index.html
o
The
Office of Inspector General’s Exclusion Program: http://oig.hhs.gov/fraud/exclusions.html
Ø Maryland Real Estate Values:
o Searchable database: http://sdatcert3.resiusa.org/rp_rewrite/
International issues:
Ø
World
Trade Organization: http://www.wto.org/english/res_e/statis_e/statis_e.htm
Ø
United
Nations Statistics Division – Social Indicators: http://unstats.un.org/unsd/demographic/social/default.htm
Ø
Ø
International
Trade Administration (
Ø
Organisation for
Economic Co-operation and Development:
o
Statistics
and databases: http://www.oecd.org/statsportal/0,2639,en_2825_293564_1_1_1_1_1,00.html
Ø
Government
of
o
Planning
Commission: http://planningcommission.nic.in/data/dataf.htm
o
Ministry
of Statistics and Programme Implementation:
·
Home
page: http://mospi.nic.in/
·
Economic
Census, 1998: http://mospi.nic.in/mospi_ec.htm
o
Election
Commission of
(includes links to data in zipped Access files)
Law enforcement, courts and crime:
Ø
o
http://www.ojp.usdoj.gov/bjs/dtd.htm
o http://www.ojp.usdoj.gov/bjs/dtdata.htm#index
Ø Federal Bureau of Investigation – Uniform Crime Reports: http://www.fbi.gov/ucr/ucr.htm
Ø
National
Archive of Criminal Justice Data: www.icpsr.umich.edu/NACJD/
Ø Office of Postsecondary Education Campus Security Statistics: http://ope.ed.gov/security/
Ø Maryland Judiciary Case Search [search engine]: http://casesearch.courts.state.md.us/inquiry/inquiry-index.jsp
Military
Ø “Windfalls of War” – The Center for Public Integrity’s report on military contractors: http://www.publicintegrity.org/wow/
Ø Department of Veterans Affairs – http://www.va.gov/vetdata/index.htm
Ø Federation of American Scientists – Arms Sales Monitoring Project: http://www.fas.org/asmp/profiles/index.html
o Sample data set from queries:
·
·
Politics:
Ø Federal Election Commission
o Downloadable databases: http://www.fec.gov/finance/disclosure/ftp_download.shtml
· Detailed files: http://www.fec.gov/finance/disclosure/ftpdet.shtml#a2005_2006
· Summary files: http://www.fec.gov/finance/disclosure/ftpsum.shtml
· Tutorial on importing FEC data into Microsoft Access: http://www.fec.gov/finance/disclosure/working_with_data_files.pdf
o Searchable databases:
· Individual contributors: http://www.fec.gov/finance/disclosure/norindsea.shtml
Ø
Ø
“The
Buying of the President 2004” – The Center for Public Integrity’s report on
candidate finances: http://www.bop2004.org/bop2004/dw.aspx
Ø “Follow the Money” – The Institute on Money
in State Politics: http://www.followthemoney.org/database/power_search.phtml
Ø The Center for Responsive Politics: www.opensecrets.org
Real estate:
Ø Maryland Department of Assessment and Taxation -- Real Property (real estate) Data Search: http://sdatcert3.resiusa.org/rp_rewrite/
Science and Technology:
Ø National Science Foundation – Division of Science Resources Statistics: http://www.nsf.gov/sbe/srs/stats.htm
Ø UNESCO Institute for Statistics: http://www.uis.unesco.org/ev.php?URL_ID=5275&URL_DO=DO_TOPIC&URL_SECTION=201
Sports and recreation
Ø Hunting accidents – Maryland Natural Resources Police: http://www.dnr.state.md.us/nrp/huntingaccidents.html
Ø Sports salaries – USA Today database:
o Baseball: http://asp.usatoday.com/sports/baseball/salaries/default.aspx
o Basketball: http://asp.usatoday.com/sports/basketball/nba/salaries/default.aspx
o Football: http://asp.usatoday.com/sports/football/nfl/salaries/default.aspx?Loc=Vanity
o Hockey: http://asp.usatoday.com/sports/hockey/nhl/salaries/default.aspx
Transportation:
Ø National Highway Traffic Safety Administration:
o Fatality
Analysis Reporting System (
o Office of Defects Investigations: http://www-odi.nhtsa.dot.gov/cars/problems/defect/defectsearch.cfm
Ø Office
of Hazardous Materials Safety,
Ø Federal Rail Administration – Office of Safety Analysis: http://safetydata.fra.dot.gov/officeofsafety/Downloads/Default.asp
Ø Bureau of Transportation Statistics – National Transportation Atlas database: http://www.bts.gov/publications/national_transportation_atlas_database/2006/
Ø Intermodal Transportation Database: www.transtats.bts.gov
Ø National Transportation Safety Board - Aviation Accident Database: http://www.ntsb.gov/NTSB/query.asp (MS Access data)
Ø Boating accidents – Maryland Natural Resources Police: http://www.dnr.state.md.us/nrp/boatingaccidents.html
Ø Federal Transit Administration – National Transit Database: http://www.ntdprogram.com/NTD/ntdhome.nsf/?Open
Ø
Mechanics of data importing
Help with data importing and data formats:
Ø Tip sheet on data formats (Jeff South, VCU): Link: (use class username and password)
Ø Tip sheets on importing text files containing data, such as files ending in *.txt, *.csv, *.dat, *.tab (you can also use the Help system in Excel and Access, and both programs have data import wizards; the tip sheets below explain how to get to the wizards and give you anoverview):
o Importing into Excel (for record sets no larger than 65,536 rows, including field names and totals)
· Overview of import wizard: http://www.edmond.k12.ok.us/online_tutorials/tutorials/excel_2002_a/excel_2002_advanced_manual_educational_webp4.htm
· Importing delimited files: http://biology.wsc.ma.edu/biology/experiments/exceldownload.html
o Importing into Access:
· Overview of importing delimited data: http://www.billianshealthdata.com/Dispstpg.htm?CD=2&ID=51
Ø SAS System Viewer – free application for viewing and printing SAS files: http://www.sas.com/apps/demosdownloads/setupcat.jsp;jsessionid=967763786440F8E958BC0E4F884805EE.tomcat4?cat=SAS+System+Viewer
Ø Converting from PDF to TEXT files: Sometimes data appears in PDF files and cannot be directly imported into database or spreadsheet programs as they are. However, depending on the way the PDF file was created, it may be possible to convert its contents into a text file that can then be imported into Excel or Access. You can read more about this here in a tip sheet from the National Institute for Computer-Assisted Reporting : http://www.ire.org/training/nettour/pdf/PDFTOTEXT.pdf