I would also be interested in open data sets. Kerrie [Quoted text hidden] Kerrie Aman Carfagno [Quoted text hidden] Susannah Dyen Tue, Mar 5, 2013 at 10:25 AM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com I would also be interested. Best, Susannah [Quoted text hidden] Simcha Levental Tue, Mar 5, 2013 at 12:08 PM Reply-To: crisismappers@googlegroups.com To: crisismappers I don't think their is one authoritative source. It's time that we embark on a effort to creating a resources spreadsheet that can assist once data is needed. This also might help us recognize data gaps that might be addressed through a data philanthropy effort. I would be happy to lead this effort. [Quoted text hidden] Andrew Turner Tue, Mar 5, 2013 at 12:50 PM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com As Simcha said there is no one-stop-shop nor authoritative source and there never will be. The solution is data federation. There will be many sites and services for open-data that relate to disaster response, both directly and indirectly, that are owned, managed, and provided from many agencies, organizations and companies. In the US there is the 'official' HSIP (Homeland Security Infrastructure Program) that has all of the national infrastructure (roads, bridges et al) and is an aggregation of mostly commercial and some government sources - but then you also have OpenStreetMap. Sites like GeoCommons, Socrata, ArcGIS Online, CKAN et al. are all hosting different official and unofficial data sources. 'BigML' has a list of lists of thousands of data repositories [1] There have also been a dozen spreadsheets and google docs to collate these. Many are event-based and short-lived and too easily forgotten about or locked away behind an opaque URL. I think a better approach is to provide loosely coupled ways to bookmark and share these repositories. Imagine if we all use Pinboard or Delicious to tag repositories and categories that we can subscribe via RSS or API's to pull together. I'm also hopeful that we can use lightweight data standards to promote federated search and data sharing. For example GeoCommons provides OpenSearch + Atom feeds that it uses itself to search remote repositories and pull in data. Anyone can support these particular standards, or others like CSW, DCAT or more. Again, more options that you can simply collate together as a simple exercise [2] This is a valuable and important problem - but let's not try to crown a single repository or create yet another spreadsheet. A good discussion on what you're trying to achieve, and ways that work in the broad community as well as for responders and agencies is far more enlightening so that we can evolve a real solution. Andrew [1] BigML Repository list: http://blog.bigml.com/2013/02/28/data-data-data-thousands-of-public-data-sources/ [2] CivicCommons Data Standards: http://wiki.civiccommons.org/Data_Standards [Quoted text hidden] -- Andrew Turner t: @ajturner b: http://highearthorbit.com m: 248.982.3609 [Quoted text hidden] Patrick Meier (iRevolution) Tue, Mar 5, 2013 at 1:02 PM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com Thanks very much, Andrew and Simcha! I love the idea of a Pinterest for open (disaster) data. Anyone know if the OpenDRI data is readily accessible? In terms of what we (QCRI & MIT/CSAIL) are trying to achieve: we're looking for linked data to pull into this free & open source DIY disaster response app. This project is still at the very early stages. Thanks again for your replies and for any additional guidance you may be able to provide. My very best, Patrick [Quoted text hidden] Andrew Turner Tue, Mar 5, 2013 at 1:14 PM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com Probably best would be to setup the pattern where data could be slotted in for a specific event. Kind of like a surge protector that you plug into the wall, then the data all can match any of N interfaces to provide: Infrastructure, People, Evacuation, Centers. That way it can adapt to whatever the most trustworthy/recent data that exist for that event. Take for example my experience sitting at FEMA during Sandy. Within a few hours FEMA went through 3 different sources for 'gas availability' - from the HSIP official, to the daily update based on credit-card data, to the IMSOCIO updating feeds. This would have been impossible to identify before-hand. #flexibility Andrew [Quoted text hidden] John Crowley Tue, Mar 5, 2013 at 4:10 PM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com Patrick, Andrew, Simcha, There is a much afoot in this area, and I'm working with a range of actors on this problem (officially and unofficially). Andrew is right about federation instead of centralized repositories, and I would like to expand on why. There are several concepts to balance here: mandate, format, and releasability. # Mandate The key here is to understand the mandates of various organizations. As Andrew points out, open data for disasters tends to be less a thing with a repository than a federation of data sources, where curation of the data remains the responsibility of the organization that generated it. That is probably the right structure. In US disaster response domestically, there is a complex method to determine which agency is the lead federal agency (LFA). Much of the time, it is FEMA/DHS. But for a chemical spill it would be EPA. And for a major terrorist incident, there would be parts of the response that would belong to different agencies (think FBI, NORTHCOM/DOD, FEMA, etc). Each will have its own implementation of Whitehouse policy on open data and its own platform for exchanging data. And one must not forget that supporting federal agencies have their own interfaces (like NGA, which is always a supporting federal agency). Also remember that much of the national response system relies on voluntary cooperation of industry and NGOs. They will each have their own method and policies of sharing operational data. A central repository would not be politically viable. International operations are bit more complex, as they tend to involve USAID/OFDA as the lead federal agency, unless it is a combat operation, in which case it is DOD. Open data is something that is up and coming, but not easy to explain in a quick email. Suffice it to say the State Dept's Humanitarian Info Unit works more magic that anyone knows--or may ever know--in making available data available to Crisis Mappers. # Format Some data may be open (public), but not use open data schema. We are well acquainted with the plague of proprietary formats and opaque, undocumented APIs, so I'll leave it at that. # Releasabilty Some data may be in open formats and curated by a lead federal agency, but may not be releasable. There are two parts to that equation: classification and caveat. Classification is a security rating: Unclassified, Secret, Top Secret, etc. Beyond UNCLAS, the classification limits access to those who have security clearances. Caveat is a second type of limitation: For Official Use Only (FOUO), No Foreigners (NOFOR), Limited Distribution (LIMDIS), etc. Data may be unclassified (UNCLAS), but marked with a caveat and therefore not releasable. The DHS HSIP dataset that Andrew mentioned has parts that are unclassified, but I believe it all is marked FOUO and some segments may have additional caveats. *I do not believe it can be made available for commercial use.* Many governments have such systems for limiting access to data. Not all are compatible, even between allies. Even for units that are in joint combat operations. And yes, that is not entirely rational. Ask a solider. Also, to my knowledge, the World Bank GFDRR OpenDRI data do not cover any US data. Hope this helps, - John [Quoted text hidden] [Quoted text hidden] Peter Burgess Tue, Mar 5, 2013 at 4:12 PM To: CrisisMappers I have been of the view that getting basic data AFTER a disaster is a bad solution to the lack of data problem. In my view society has the need for such data all the time ... but of course these data are largely missing. I grew up in England. Details of Okehampton, the little town where I grew up are in the Domesday Book, an inventory of important data about England compiled by William after his conquest of England in 1066. I argue that technology has improved substantially since then and there should be some way for important data to be easily available about everywhere all the time. Some of these data are emerging, and some of what Google has done seems pretty impressive. There are all sorts of social issues including privacy and the value of these data for emergency response. I would argue that such data could also have an important role in transparency, accounting and accountability, monitoring and evaluation (TAAME) which should in my view be much more about impact in a place (and the society in the place) than impact in an organization. The data needed for emergency response are a subset of the data I argue are routinely needed to get better progress and performance in society. In fact, because these data are not available for society ... though they often are available inside the corporate for profit fire wall ... corporate organizations are able to optimize performance and society all too often gets screwed. I followed the data flows after the Haiti earthquake. One of my lasting impressions was how much data lived a life of its own, completely irrelevant to the issues needing to be addressed on the ground. This is a problem where there is operational inexperience ... an increasing problem in all of society. Maybe one can argue that much of modern society is in a permanent state of (economic) disaster and it would be good to deploy meaningful data now as much as possible. What might I be able to do to help/ Peter Burgess [Quoted text hidden] -- ____________ Peter Burgess TrueValueMetrics ... Meaningful Metrics for a Smart Society twitter: @truevaluemetric @peterbnyc www.truevaluemetrics.org blog: http://truevaluemetrics.blogspot.com blog: http://communityanalyticsca.blogspot.com mobile: 212 744 6469 email: peterbnyc@gmail.com skype: peterburgessnyc Books: Search Peter Burgess at www.lulu.com Jon Nystrom Wed, Mar 6, 2013 at 7:55 AM Reply-To: crisismappers@googlegroups.com To: '' Cc: 'crisismappers@googlegroups.com' John, From the HSIP perspective there is Freedom and Gold. Gold has licensing restrictions and Freedom should be available to parties that need it with no restrictions. Other places people may want to look are: HSIN - if you have mandate and right credentials. Data.gov Geoplatform.gov FEMA also has open services that they stand up during events for their partners and the public. Jon Sent from my iPhone [Quoted text hidden] [Quoted text hidden] Kirk Wed, Mar 6, 2013 at 10:04 AM Reply-To: crisismappers@googlegroups.com To: CrisisMappers I had been trying to encourage some to work with Greg Matthews of US Geological Survey within the National Geospatial Technical Operations Center (NGTOC). The National Map Corp http://nationalmap.gov/TheNationalMapCorps/ could build the data sets you are looking for by crowdsouring means. All data created by the Corps will be opendata. km On Mar 5, 8:04 am, 'Patrick Meier (iRevolution)' [Quoted text hidden] Patrick Florance Wed, Mar 6, 2013 at 3:08 PM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com Cc: patrick@irevolution.net Hello, The Open Geoportal project certainly falls in this domain. The Open Geoportal (OGP) is a collaboratively developed, open source, federated web application to rapidly discover, preview, and retrieve geospatial data from multiple repositories. It is lead by Tufts University along with Harvard and MIT. There are currently over 30 institutional and organization partners. The data ranges from global to municipal scale and extents and is both international and domestic. We will be ingesting many national web services shortly. Here is an instance: http://geodata.tufts.edu -Just zoom to Boston then to San Francisco then to a state like Massachusetts to see how it works. You'll see some Tufts restricted data in there but much of it is open data. We also have a new organizational site that is still in beta mode but going live in two weeks that provides more information about Open Geoportal http://opengeoportal.org I'd be happy to discuss with anyone interested. We have many new developments coming in the next couple months. Best, Patrick Patrick Florance Manager of Geospatial Technology Services, Tufts Technology Services Lecturer, Fletcher School of International Law & Diplomacy Tufts University 16 Dearborn Rd. Somerville, MA 02144 phone: 617-627-4235 email: patrick.florance@tufts.edu http://gis.tufts.edu Skype: patrick.florance On Tuesday, March 5, 2013 8:04:08 AM UTC-5, Patrick Meier wrote: Dear CrisisMappers, Does anyone know of a one-stop-shop repository of open datasets useful for disaster response in the US? Thank you, Patrick [Quoted text hidden] Patrick Florance Wed, Mar 6, 2013 at 4:15 PM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com Cc: patrick@irevolution.net Just adding a little more. I agree with all that's been posted and federation along with high quality search, regardless of the portal interface, is key. We've been focusing heavily on rapid search and acquisition. Andrew touched on an area that we would like to pursue, that is, rapid discover/acquisition of the most 'relevant' data which is a combination of feature type, date, scale, extent, coverage, attribution, quality metadata, data duplication, etc. We'd be really interested in discussing efforts and approaches here. Patrick [Quoted text hidden] Christiaan Adams Wed, Mar 6, 2013 at 5:42 PM Reply-To: crisismappers@googlegroups.com To: crisismappers Cc: Patrick Meier The Google Earth Gallery contains a collection of datasets for viewing in Google Earth and/or Google Maps. Most of them are in KML format (click the 'View in Google Earth' button to download the KML), or are served as Google Maps Engine layers (see the links on each map page for Maps API codes, an embeddable widget, and a WMS link). The Gallery includes a section on Crisis & Humanitarian datasets, here: http://www.google.com/gadgets/directory?synd=earth&cat=crisis_resp That category currently contains several dozen layers, about half of them crisis related, from various providers. Many of them are layers we've used on our crisis maps for disaster events. It's currently far from a comprehensive collection, but we're definitely hoping to get more content in there, and to help make it easy to find and use. For users of Google Maps Engine, here's info on submitting your maps to the Gallery: http://support.google.com/mapsengine/bin/answer.py?hl=en&answer=2780658 Cheers, -Christiaan ---------------------------------- Christiaan Adams Google Earth Outreach earth.google.com/outreach Google.org Crisis Response google.org/crisisresponse Random Hacks of Kindness rhok.org ---------------------------------- [Quoted text hidden] terra@citizenactionteam.org Thu, Mar 7, 2013 at 12:09 AM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com, Patrick Florance Cc: patrick@irevolution.net I'm sorry I'm late to this conversation. We have the www.citizencommandcenter.org/quick/start database in CSV format finally. One time...Not just disaster relief. And not up to date for every disaster. Usually by the time we're done posting on a disaster, we're so exhausted that we sleep...then another disaster hits and so we move on to that one and just leave the other stuff as 'standby', best we can. Anyway, we don't have a way for people to pull the data yet live, but we will hopefully with this new round of coders...they look really adept. The plan is for people to be able to pull it with some kind of way to track who got it, in case we find that the data was used for nefarious reasons. But til that's available, you can write to me and I'll send you the data. YOu'll have to swear to abide by ethical treatment of data. For faster, response while I'm in this migration period (til end of March), please write to terrafATcompuserveDOTcom. 978 808 7173 sorry it's not open yet, our coding staff has been atnegative one for a couple years, aggravated by my commitment to revolution... [Quoted text hidden] ---You received this message because you are subscribed to the Google Groups 'CrisisMappers' group. [Quoted text hidden] terra@citizenactionteam.org Thu, Mar 7, 2013 at 12:14 AM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com, Christiaan Adams Cc: crisismappers , Patrick Meier Excellent. I hope its the actual data, cuz the last time I tried to track it down, I only got graphics. No time to look at this now, but will do. in the CSV we've got, if you're interested in humanitarian services in general, mixed w/disaster relief, we have over 8,000 orgs listed...well, a variety of orgs and reference points...pointers to other directories (no point in re-entereing stuff)... and status records (lists of whatever useful stuff) om, is this what you meant? if so, yeah, we have the best list of haiti relief sites, for example. or so someone from the UN told us last year. not sure what's happened since, though. so maybe it's old. last time we thought our haiti data sucked someone else came and said it was the best. t Quoting Christiaan Adams : The Google Earth Gallery contains a collection of datasets for viewing in Google Earth and/or Google Maps. Most of them are in KML format (click the 'View in Google Earth' button to download the KML), or are served as Google Maps Enginelayers (see the links on each map page for Maps API codes, an embeddable widget, and a WMS link). The Gallery includes a section on *Crisis & Humanitarian* datasets, here: http://www.google.com/gadgets/directory?synd=earth&cat=crisis_resp That category currently contains several dozen layers, about half of them crisis related, from various providers. Many of them are layers we've used on our crisis maps for disaster events. It's currently far from a comprehensive collection, but we're definitely hoping to get more content in there, and to help make it easy to find and use. For users of Google Maps Engine, here's info on submitting your maps to the Gallery: http://support.google.com/mapsengine/bin/answer.py?hl=en&answer=2780658 Cheers, -Christiaan ---------------------------------- Christiaan Adams Google Earth Outreach earth.google.com/outreach Google.org Crisis Response google.org/crisisresponse Random Hacks of Kindness rhok.org [Quoted text hidden] [Quoted text hidden] [Quoted text hidden] Tim McNamara Thu, Mar 7, 2013 at 1:39 AM Reply-To: crisismappers@googlegroups.com To: CrisisMappers Cc: Christiaan Adams , Patrick Meier Hi all, datahub.io allows you to upload data, provide structured metadata and create curated groups. It could be a good base. [Quoted text hidden] [Quoted text hidden] terra@citizenactionteam.org Fri, Mar 8, 2013 at 2:49 AM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com, Tim McNamara Cc: CrisisMappers , Christiaan Adams , Patrick Meier Tim, This might be an interesting way to save us the time of building an Open API... I'm finding that it's not as easy as I'd thought. I'll be starting a new thread to ask some questions about that. With respect to the platform below...thanks for the heads up. We have not started sharing automatically. And I'd like to do so. Maybe this is the way, instead of building an Open API? Terra Quoting Tim McNamara : Hi all, datahub.io allows you to upload data, provide structured metadata and create curated groups. It could be a good base. On 7 March 2013 18:14, wrote: Excellent. I hope its the actual data, cuz the last time I tried to track it down, I only got graphics. No time to look at this now, but will do. in the CSV we've got, if you're interested in humanitarian services in general, mixed w/disaster relief, we have over 8,000 orgs listed...well, a variety of orgs and reference points...pointers to other directories (no point in re-entereing stuff)... and status records (lists of whatever useful stuff) om, is this what you meant? if so, yeah, we have the best list of haiti relief sites, for example. or so someone from the UN told us last year. not sure what's happened since, though. so maybe it's old. last time we thought our haiti data sucked someone else came and said it was the best. t Quoting Christiaan Adams : The Google Earth Gallery> contains a collection of datasets for viewing in Google Earth and/or Google Maps. Most of them are in KML format (click the 'View in Google Earth' button to download the KML), or are served as Google Maps Engine >layers (see the links on each map page for Maps API codes, an embeddable widget, and a WMS link). The Gallery includes a section on *Crisis & Humanitarian* datasets, here: http://www.google.com/gadgets/**directory?synd=earth&cat=**crisis_resp That category currently contains several dozen layers, about half of them crisis related, from various providers. Many of them are layers we've used on our crisis maps for disaster events. It's currently far from a comprehensive collection, but we're definitely hoping to get more content in there, and to help make it easy to find and use. For users of Google Maps Engine, here's info on submitting your maps to the Gallery: http://support.google.com/**mapsengine/bin/answer.py?hl=** en&answer=2780658 Cheers, -Christiaan ------------------------------**---- Christiaan Adams Google Earth Outreach earth.google.com/outreach Google.org Crisis Response google.org/crisisresponse > Random Hacks of Kindness rhok.org [Quoted text hidden] [Quoted text hidden] terra@citizenactionteam.org Fri, Mar 8, 2013 at 3:00 AM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com, Jon Nystrom Cc: '' Thanks, Jon... Can you give me more info about FEMA? I have not seen them stand up anything that is useful, except for their DRC locations. We gave our data to HSIN and didn't get anything back. I wasn't impressed with their work. Maybe my opinion is old. Will look at the others. Terra Quoting Jon Nystrom : John, From the HSIP perspective there is Freedom and Gold. Gold has licensing restrictions and Freedom should be available to parties that need it with no restrictions. Other places people may want to look are: HSIN - if you have mandate and right credentials. Data.gov Geoplatform.gov FEMA also has open services that they stand up during events for their partners and the public. Jon Sent from my iPhone On Mar 5, 2013, at 2:10 PM, 'John Crowley' > wrote: Patrick, Andrew, Simcha, There is a much afoot in this area, and I'm working with a range of actors on this problem (officially and unofficially). Andrew is right about federation instead of centralized repositories, and I would like to expand on why. There are several concepts to balance here: mandate, format, and releasability. # Mandate The key here is to understand the mandates of various organizations. As Andrew points out, open data for disasters tends to be less a thing with a repository than a federation of data sources, where curation of the data remains the responsibility of the organization that generated it. That is probably the right structure. In US disaster response domestically, there is a complex method to determine which agency is the lead federal agency (LFA). Much of the time, it is FEMA/DHS. But for a chemical spill it would be EPA. And for a major terrorist incident, there would be parts of the response that would belong to different agencies (think FBI, NORTHCOM/DOD, FEMA, etc). Each will have its own implementation of Whitehouse policy on open data and its own platform for exchanging data. And one must not forget that supporting federal agencies have their own interfaces (like NGA, which is always a supporting federal agency). Also remember that much of the national response system relies on voluntary cooperation of industry and NGOs. They will each have their own method and policies of sharing operational data. A central repository would not be politically viable. International operations are bit more complex, as they tend to involve USAID/OFDA as the lead federal agency, unless it is a combat operation, in which case it is DOD. Open data is something that is up and coming, but not easy to explain in a quick email. Suffice it to say the State Dept's Humanitarian Info Unit works more magic that anyone knows--or may ever know--in making available data available to Crisis Mappers. # Format Some data may be open (public), but not use open data schema. We are well acquainted with the plague of proprietary formats and opaque, undocumented APIs, so I'll leave it at that. # Releasabilty Some data may be in open formats and curated by a lead federal agency, but may not be releasable. There are two parts to that equation: classification and caveat. Classification is a security rating: Unclassified, Secret, Top Secret, etc. Beyond UNCLAS, the classification limits access to those who have security clearances. Caveat is a second type of limitation: For Official Use Only (FOUO), No Foreigners (NOFOR), Limited Distribution (LIMDIS), etc. Data may be unclassified (UNCLAS), but marked with a caveat and therefore not releasable. The DHS HSIP dataset that Andrew mentioned has parts that are unclassified, but I believe it all is marked FOUO and some segments may have additional caveats. *I do not believe it can be made available for commercial use.* Many governments have such systems for limiting access to data. Not all are compatible, even between allies. Even for units that are in joint combat operations. And yes, that is not entirely rational. Ask a solider. Also, to my knowledge, the World Bank GFDRR OpenDRI data do not cover any US data. Hope this helps, - John Andrew Turner wrote: Probably best would be to setup the pattern where data could be slotted in for a specific event. Kind of like a surge protector that you plug into the wall, then the data all can match any of N interfaces to provide: Infrastructure, People, Evacuation, Centers. That way it can adapt to whatever the most trustworthy/recent data that exist for that event. Take for example my experience sitting at FEMA during Sandy. Within a few hours FEMA went through 3 different sources for 'gas availability' - from the HSIP official, to the daily update based on credit-card data, to the IMSOCIO updating feeds. This would have been impossible to identify before-hand. #flexibility Andrew On Tue, Mar 5, 2013 at 1:02 PM, Patrick Meier (iRevolution) > wrote: Thanks very much, Andrew and Simcha! I love the idea of a Pinterest for open (disaster) data. Anyone know if the OpenDRI data is readily accessible? In terms of what we (QCRI & MIT/CSAIL) are trying to achieve: we're looking for linked data to pull into this free & open source DIY disaster response app. This project is still at the very early stages. Thanks again for your replies and for any additional guidance you may be able to provide. My very best, Patrick On Tue, Mar 5, 2013 at 12:50 PM, Andrew Turner > wrote: As Simcha said there is no one-stop-shop nor authoritative source and there never will be. The solution is data federation. There will be many sites and services for open-data that relate to disaster response, both directly and indirectly, that are owned, managed, and provided from many agencies, organizations and companies. In the US there is the 'official' HSIP (Homeland Security Infrastructure Program) that has all of the national infrastructure (roads, bridges et al) and is an aggregation of mostly commercial and some government sources - but then you also have OpenStreetMap. Sites like GeoCommons, Socrata, ArcGIS Online, CKAN et al. are all hosting different official and unofficial data sources. 'BigML' has a list of lists of thousands of data repositories [1] There have also been a dozen spreadsheets and google docs to collate these. Many are event-based and short-lived and too easily forgotten about or locked away behind an opaque URL. I think a better approach is to provide loosely coupled ways to bookmark and share these repositories. Imagine if we all use Pinboard or Delicious to tag repositories and categories that we can subscribe via RSS or API's to pull together. I'm also hopeful that we can use lightweight data standards to promote federated search and data sharing. For example GeoCommons provides OpenSearch + Atom feeds that it uses itself to search remote repositories and pull in data. Anyone can support these particular standards, or others like CSW, DCAT or more. Again, more options that you can simply collate together as a simple exercise [2] This is a valuable and important problem - but let's not try to crown a single repository or create yet another spreadsheet. A good discussion on what you're trying to achieve, and ways that work in the broad community as well as for responders and agencies is far more enlightening so that we can evolve a real solution. Andrew [1] BigML Repository list: http://blog.bigml.com/2013/02/28/data-data-data-thousands-of-public-data-sources/ [2] CivicCommons Data Standards: http://wiki.civiccommons.org/Data_Standards On Tue, Mar 5, 2013 at 12:08 PM, Simcha Levental > wrote: I don't think their is one authoritative source. It's time that we embark on a effort to creating a resources spreadsheet that can assist once data is needed. This also might help us recognize data gaps that might be addressed through a data philanthropy effort. I would be happy to lead this effort. On Mar 5, 2013 10:25 AM, 'Susannah Dyen' > wrote: I would also be interested. Best, Susannah On Tue, Mar 5, 2013 at 9:47 AM, Kerrie Aman Carfagno > wrote: I would also be interested in open data sets. Kerrie On Mar 5, 2013, at 8:04 AM, Patrick Meier (iRevolution) wrote: Dear CrisisMappers, Does anyone know of a one-stop-shop repository of open datasets useful for disaster response in the US? Thank you, Patrick Kerrie Aman Carfagno -- Andrew Turner t: @ajturner b: http://highearthorbit.com m: 248.982.3609 -- Andrew Turner t: @ajturner b: http://highearthorbit.com m: 248.982.3609 terra@citizenactionteam.org Fri, Mar 8, 2013 at 3:01 AM To: crisismappers@googlegroups.com, Peter Burgess Cc: CrisisMappers People don't want to tell you that they have a chainsaw before a disaster. But when there IS a disaster in their region, then they will tell you that they have a chainsaw. Terra Quoting Peter Burgess : [Quoted text hidden] [Quoted text hidden] [Quoted text hidden] disaster response app. This project is still at the *very* early stages. [Quoted text hidden] [Quoted text hidden] [Quoted text hidden] terra@citizenactionteam.org Fri, Mar 8, 2013 at 12:06 PM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com, John Crowley FEMA/VOADs have built a structure for response, including various 'typical roles', like 'mass care' (feeding, etc). Groups determine which response team they are on. Red Cross is in mass care, I believe. Anyway, my issue is not with this, which didn't exist to any comprehensive degree (it was loose and not followed) even as far back as a few years ago. My issue is that no one agrees to or abides by any performance metrics. So the RC may SAY that they will get a mobile food canteen out to some neighborhood, but nothing stops them from blowing it off. And they do. Nothing against them, in particular. All of the groups are loosy goosy as to their willingness to commit and follow through consistently. And it's nothing 'against them'...they try. The people try. People care. It's just that we, as a public, have never held them to any standards. And so there are none. And failure is an acceptable response in 'disaster times'. That's where people like us come in. We track the needs, focusing on the areas that the big guys aren't addressing, find the sites that need, find the groups that have, connect the two, then follow up to make sure the need has been met. This is one of the reasons that the SMS system doesn't do much for me. There isnt' anyone committing to addressing the actual alerts. I've given the attached to this list a couple of times, and no one has commented. But we need to have all systems fall into specific roles, rather than every system trying to do 'everything'. I have ideas on what those roles might be. I'm a systems integrator who built command and control systems before I realized that war is not the answer. Actually, I always knew that. But I had thought that we could create war that didn't require bloodshed. Silly me. Anyway, more later. Gotta run... Terra Quoting John Crowley : Patrick, Andrew, Simcha, There is a much afoot in this area, and I'm working with a range of actors on this problem (officially and unofficially). Andrew is right about federation instead of centralized repositories, and I would like to expand on why. There are several concepts to balance here: mandate, format, and releasability. # Mandate The key here is to understand the mandates of various organizations. As Andrew points out, open data for disasters tends to be less a thing with a repository than a federation of data sources, where curation of the data remains the responsibility of the organization that generated it. That is probably the right structure. In US disaster response domestically, there is a complex method to determine which agency is the lead federal agency (LFA). Much of the time, it is FEMA/DHS. But for a chemical spill it would be EPA. And for a major terrorist incident, there would be parts of the response that would belong to different agencies (think FBI, NORTHCOM/DOD, FEMA, etc). Each will have its own implementation of Whitehouse policy on open data and its own platform for exchanging data. And one must not forget that supporting federal agencies have their own interfaces (like NGA, which is always a supporting federal agency). Also remember that much of the national response system relies on voluntary cooperation of industry and NGOs. They will each have their own method and policies of sharing operational data. A central repository would not be politically viable. International operations are bit more complex, as they tend to involve USAID/OFDA as the lead federal agency, unless it is a combat operation, in which case it is DOD. Open data is something that is up and coming, but not easy to explain in a quick email. Suffice it to say the State Dept's Humanitarian Info Unit works more magic that anyone knows--or may ever know--in making available data available to Crisis Mappers. # Format Some data may be open (public), but not use open data schema. We are well acquainted with the plague of proprietary formats and opaque, undocumented APIs, so I'll leave it at that. # Releasabilty Some data may be in open formats and curated by a lead federal agency, but may not be releasable. There are two parts to that equation: classification and caveat. Classification is a security rating: Unclassified, Secret, Top Secret, etc. Beyond UNCLAS, the classification limits access to those who have security clearances. Caveat is a second type of limitation: For Official Use Only (FOUO), No Foreigners (NOFOR), Limited Distribution (LIMDIS), etc. Data may be unclassified (UNCLAS), but marked with a caveat and therefore not releasable. The DHS HSIP dataset that Andrew mentioned has parts that are unclassified, but I believe it all is marked FOUO and some segments may have additional caveats. *I do not believe it can be made available for commercial use.* Many governments have such systems for limiting access to data. Not all are compatible, even between allies. Even for units that are in joint combat operations. And yes, that is not entirely rational. Ask a solider. Also, to my knowledge, the World Bank GFDRR OpenDRI data do not cover any US data. Hope this helps, - John Andrew Turner wrote: Probably best would be to setup the pattern where data could be slotted in for a specific event. Kind of like a surge protector that you plug into the wall, then the data all can match any of N interfaces to provide: Infrastructure, People, Evacuation, Centers. That way it can adapt to whatever the most trustworthy/recent data that exist for that event. Take for example my experience sitting at FEMA during Sandy. Within a few hours FEMA went through 3 different sources for 'gas availability' - from the HSIP official, to the daily update based on credit-card data, to the IMSOCIO updating feeds. This would have been impossible to identify before-hand. #flexibility Andrew On Tue, Mar 5, 2013 at 1:02 PM, Patrick Meier (iRevolution) > wrote: Thanks very much, Andrew and Simcha! I love the idea of a Pinterest for open (disaster) data. Anyone know if the OpenDRI data is readily accessible? In terms of what we (QCRI & MIT/CSAIL) are trying to achieve: we're looking for linked data to pull into this free & open source DIY disaster response app . This project is still at the /very/ early stages. Thanks again for your replies and for any additional guidance you may be able to provide. My very best, Patrick On Tue, Mar 5, 2013 at 12:50 PM, Andrew Turner > [Quoted text hidden] > wrote: I don't think their is one authoritative source. It's time that we embark on a effort to creating a resources spreadsheet that can assist once data is needed. This also might help us recognize data gaps that might be addressed through a data philanthropy effort. I would be happy to lead this effort. On Mar 5, 2013 10:25 AM, 'Susannah Dyen' > wrote: I would also be interested. Best, Susannah On Tue, Mar 5, 2013 at 9:47 AM, Kerrie Aman Carfagno > wrote: I would also be interested in open data sets. Kerrie On Mar 5, 2013, at 8:04 AM, Patrick Meier (iRevolution) wrote: Dear CrisisMappers, Does anyone know of a one-stop-shop repository of open datasets useful for disaster response in the US? Thank you, Patrick Kerrie Aman Carfagno -- Andrew Turner t: @ajturner b: http://highearthorbit.com m: 248.982.3609 -- CrisisMappers | The Humanitarian Technology Network http://www.CrisisMappers.net Disaster Response Tracking Rev 0.pdf 3590K Andrew Turner Sat, Mar 9, 2013 at 7:01 AM Reply-To: crisismappers@googlegroups.com To: 'crisismappers@googlegroups.com' Cc: 'crisismappers@googlegroups.com' , Jon Nystrom That's unfortunate to hear Terra. I'm reminded of 'Huffman's Principles for Data Sharing' http://blog.geoiq.com/2009/09/16/huffmans-three-principles-for-data-sharing/ 1) Create immediate value for anyone contributing data 2) Make contributor’s data available back to them with improvements 3) Share derivative works back with the data sharing community @ajturner - sent from the moon [Quoted text hidden] [Quoted text hidden] Jon Nystrom Sat, Mar 9, 2013 at 8:08 AM Reply-To: crisismappers@googlegroups.com To: Andrew Turner , 'terra@citizenactionteam.org' Cc: 'crisismappers@googlegroups.com' I would say they are several services that FEMA stood up that proved to be useful: Modeling Task Force, that they were updating on almost a daily basis: http://fema.maps.arcgis.com/home/item.html?id=307dd522499d4a44a33d7296a5da5ea0 Check Your Home Application - powered by images collected from Civil Air Patrol: http://fema.maps.arcgis.com/home/item.html?id=a49076c0623249ef8d3adad74ca93096 Power Task Force: http://fema.maps.arcgis.com/home/item.html?id=cb946a364c59411cb70a57c2adda5445 DRC and Shelter: http://fema.maps.arcgis.com/home/item.html?id=937be5d9eaca451080e4eee0670945a6 Recovery phase ABFE services: http://fema.maps.arcgis.com/home/item.html?id=2f0a884bfb434d76af8c15c26541a545 There were various other services stood up including Imagery collected by NOAA and Commercial Imagery. Jon From: Andrew Turner [mailto:ajturner@gmail.com] Sent: Saturday, March 09, 2013 7:01 AM To: crisismappers@googlegroups.com Cc: crisismappers@googlegroups.com; Jon Nystrom Subject: Re: [CrisisMappers] open datasets for disaster response? [Quoted text hidden] [Quoted text hidden] Visit this group at http://groups.google.com/group/crisismappers?hl=en-US. [Quoted text hidden] terra@citizenactionteam.org Sat, Mar 9, 2013 at 7:40 PM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com, Jon Nystrom Cc: Andrew Turner , 'crisismappers@googlegroups.com' I haven't seen the ABFE services. That's interesting. Thanks for the heads up. I didn't realize they had expanded what they are doing. Very refreshing. T Quoting Jon Nystrom : [Quoted text hidden] On Mar 8, 2013, at 3:00 AM, terra@citizenactionteam.org wrote: Thanks, Jon... Can you give me more info about FEMA? I have not seen them stand up anything that is useful, except for their DRC locations. We gave our data to HSIN and didn't get anything back. I wasn't impressed with their work. Maybe my opinion is old. Will look at the others. Terra Quoting Jon Nystrom >: John, From the HSIP perspective there is Freedom and Gold. Gold has licensing restrictions and Freedom should be available to parties that need it with no restrictions. Other places people may want to look are: HSIN - if you have mandate and right credentials. Data.gov Geoplatform.gov FEMA also has open services that they stand up during events for their partners and the public. Jon Sent from my iPhone On Mar 5, 2013, at 2:10 PM, 'John Crowley' > wrote: Patrick, Andrew, Simcha, There is a much afoot in this area, and I'm working with a range of actors on this problem (officially and unofficially). Andrew is right about federation instead of centralized repositories, and I would like to expand on why. There are several concepts to balance here: mandate, format, and releasability. # Mandate The key here is to understand the mandates of various organizations. As Andrew points out, open data for disasters tends to be less a thing with a repository than a federation of data sources, where curation of the data remains the responsibility of the organization that generated it. That is probably the right structure. In US disaster response domestically, there is a complex method to determine which agency is the lead federal agency (LFA). Much of the time, it is FEMA/DHS. But for a chemical spill it would be EPA. And for a major terrorist incident, there would be parts of the response that would belong to different agencies (think FBI, NORTHCOM/DOD, FEMA, etc). Each will have its own implementation of Whitehouse policy on open data and its own platform for exchanging data. And one must not forget that supporting federal agencies have their own interfaces (like NGA, which is always a supporting federal agency). Also remember that much of the national response system relies on voluntary cooperation of industry and NGOs. They will each have their own method and policies of sharing operational data. A central repository would not be politically viable. International operations are bit more complex, as they tend to involve USAID/OFDA as the lead federal agency, unless it is a combat operation, in which case it is DOD. Open data is something that is up and coming, but not easy to explain in a quick email. Suffice it to say the State Dept's Humanitarian Info Unit works more magic that anyone knows--or may ever know--in making available data available to Crisis Mappers. # Format Some data may be open (public), but not use open data schema. We are well acquainted with the plague of proprietary formats and opaque, undocumented APIs, so I'll leave it at that. # Releasabilty Some data may be in open formats and curated by a lead federal agency, but may not be releasable. There are two parts to that equation: classification and caveat. Classification is a security rating: Unclassified, Secret, Top Secret, etc. Beyond UNCLAS, the classification limits access to those who have security clearances. Caveat is a second type of limitation: For Official Use Only (FOUO), No Foreigners (NOFOR), Limited Distribution (LIMDIS), etc. Data may be unclassified (UNCLAS), but marked with a caveat and therefore not releasable. The DHS HSIP dataset that Andrew mentioned has parts that are unclassified, but I believe it all is marked FOUO and some segments may have additional caveats. *I do not believe it can be made available for commercial use.* Many governments have such systems for limiting access to data. Not all are compatible, even between allies. Even for units that are in joint combat operations. And yes, that is not entirely rational. Ask a solider. Also, to my knowledge, the World Bank GFDRR OpenDRI data do not cover any US data. Hope this helps, - John Andrew Turner wrote: Probably best would be to setup the pattern where data could be slotted in for a specific event. Kind of like a surge protector that you plug into the wall, then the data all can match any of N interfaces to provide: Infrastructure, People, Evacuation, Centers. That way it can adapt to whatever the most trustworthy/recent data that exist for that event. Take for example my experience sitting at FEMA during Sandy. Within a few hours FEMA went through 3 different sources for 'gas availability' - from the HSIP official, to the daily update based on credit-card data, to the IMSOCIO updating feeds. This would have been impossible to identify before-hand. #flexibility Andrew On Tue, Mar 5, 2013 at 1:02 PM, Patrick Meier (iRevolution) > wrote: Thanks very much, Andrew and Simcha! I love the idea of a Pinterest for open (disaster) data. Anyone know if the OpenDRI data is readily accessible? In terms of what we (QCRI & MIT/CSAIL) are trying to achieve: we're looking for linked data to pull into this free & open source DIY disaster response app. This project is still at the very early stages. Thanks again for your replies and for any additional guidance you may be able to provide. My very best, Patrick On Tue, Mar 5, 2013 at 12:50 PM, Andrew Turner > wrote: As Simcha said there is no one-stop-shop nor authoritative source and there never will be. The solution is data federation. There will be many sites and services for open-data that relate to disaster response, both directly and indirectly, that are owned, managed, and provided from many agencies, organizations and companies. In the US there is the 'official' HSIP (Homeland Security Infrastructure Program) that has all of the national infrastructure (roads, bridges et al) and is an aggregation of mostly commercial and some government sources - but then you also have OpenStreetMap. Sites like GeoCommons, Socrata, ArcGIS Online, CKAN et al. are all hosting different official and unofficial data sources. 'BigML' has a list of lists of thousands of data repositories [1] There have also been a dozen spreadsheets and google docs to collate these. Many are event-based and short-lived and too easily forgotten about or locked away behind an opaque URL. I think a better approach is to provide loosely coupled ways to bookmark and share these repositories. Imagine if we all use Pinboard or Delicious to tag repositories and categories that we can subscribe via RSS or API's to pull together. I'm also hopeful that we can use lightweight data standards to promote federated search and data sharing. For example GeoCommons provides OpenSearch + Atom feeds that it uses itself to search remote repositories and pull in data. Anyone can support these particular standards, or others like CSW, DCAT or more. Again, more options that you can simply collate together as a simple exercise [2] This is a valuable and important problem - but let's not try to crown a single repository or create yet another spreadsheet. A good discussion on what you're trying to achieve, and ways that work in the broad community as well as for responders and agencies is far more enlightening so that we can evolve a real solution. Andrew [1] BigML Repository list: http://blog.bigml.com/2013/02/28/data-data-data-thousands-of-public-data-sources/ [2] CivicCommons Data Standards: http://wiki.civiccommons.org/Data_Standards On Tue, Mar 5, 2013 at 12:08 PM, Simcha Levental > wrote: I don't think their is one authoritative source. It's time that we embark on a effort to creating a resources spreadsheet that can assist once data is needed. This also might help us recognize data gaps that might be addressed through a data philanthropy effort. I would be happy to lead this effort. On Mar 5, 2013 10:25 AM, 'Susannah Dyen' > wrote: I would also be interested. Best, Susannah On Tue, Mar 5, 2013 at 9:47 AM, Kerrie Aman Carfagno > wrote: I would also be interested in open data sets. Kerrie On Mar 5, 2013, at 8:04 AM, Patrick Meier (iRevolution) wrote: Dear CrisisMappers, Does anyone know of a one-stop-shop repository of open datasets useful for disaster response in the US? Thank you, Patrick Kerrie Aman Carfagno -- Andrew Turner t: @ajturner b: http://highearthorbit.com m: 248.982.3609 -- CrisisMappers | The Humanitarian Technology Network http://www.CrisisMappers.net -- Andrew Turner t: @ajturner b: http://highearthorbit.com m: 248.982.3609 -- terra@citizenactionteam.org Sat, Mar 9, 2013 at 7:42 PM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com, Andrew Turner Cc: 'crisismappers@googlegroups.com' , Jon Nystrom Love this. Sounds like a reasonable 'user agreement' to me. T [Quoted text hidden] Visit this group at http://groups.google.com/group/crisismappers?hl=en-US. [Quoted text hidden] terra@citizenactionteam.org Sun, Mar 10, 2013 at 12:05 AM Reply-To: crisismappers@googlegroups.com To: crisismappers@googlegroups.com, Jon Nystrom Cc: Andrew Turner , 'crisismappers@googlegroups.com' Jon, These links don't work for me. I tried allowing cookies, and I get the main site, but not the specific site you reference. Terra Quoting Jon Nystrom : [Quoted text hidden] On Mar 8, 2013, at 3:00 AM, terra@citizenactionteam.org wrote: Thanks, Jon... Can you give me more info about FEMA? I have not seen them stand up anything that is useful, except for their DRC locations. We gave our data to HSIN and didn't get anything back. I wasn't impressed with their work. Maybe my opinion is old. Will look at the others. Terra Quoting Jon Nystrom >: John, From the HSIP perspective there is Freedom and Gold. Gold has licensing restrictions and Freedom should be available to parties that need it with no restrictions. Other places people may want to look are: HSIN - if you have mandate and right credentials. Data.gov Geoplatform.gov FEMA also has open services that they stand up during events for their partners and the public. Jon Sent from my iPhone On Mar 5, 2013, at 2:10 PM, 'John Crowley' > wrote: Patrick, Andrew, Simcha,

There is a much afoot in this area, and I'm working with a range of actors on this problem (officially and unofficially). Andrew is right about federation instead of centralized repositories, and I would like to expand on why.

There are several concepts to balance here: mandate, format, and releasability.

# Mandate

The key here is to understand the mandates of various organizations. As Andrew points out, open data for disasters tends to be less a thing with a repository than a federation of data sources, where curation of the data remains the responsibility of the organization that generated it. That is probably the right structure.

In US disaster response domestically, there is a complex method to determine which agency is the lead federal agency (LFA). Much of the time, it is FEMA/DHS. But for a chemical spill it would be EPA. And for a major terrorist incident, there would be parts of the response that would belong to different agencies (think FBI, NORTHCOM/DOD, FEMA, etc). Each will have its own implementation of Whitehouse policy on open data and its own platform for exchanging data. And one must not forget that supporting federal agencies have their own interfaces (like NGA, which is always a supporting federal agency).

Also remember that much of the national response system relies on voluntary cooperation of industry and NGOs. They will each have their own method and policies of sharing operational data. A central repository would not be politically viable.

International operations are bit more complex, as they tend to involve USAID/OFDA as the lead federal agency, unless it is a combat operation, in which case it is DOD. Open data is something that is up and coming, but not easy to explain in a quick email. Suffice it to say the State Dept's Humanitarian Info Unit works more magic that anyone knows--or may ever know--in making available data available to Crisis Mappers.

# Format Some data may be open (public), but not use open data schema. We are well acquainted with the plague of proprietary formats and opaque, undocumented APIs, so I'll leave it at that.

# Releasabilty

Some data may be in open formats and curated by a lead federal agency, but may not be releasable. There are two parts to that equation: classification and caveat. Classification is a security rating: Unclassified, Secret, Top Secret, etc. Beyond UNCLAS, the classification limits access to those who have security clearances. Caveat is a second type of limitation: For Official Use Only (FOUO), No Foreigners (NOFOR), Limited Distribution (LIMDIS), etc. Data may be unclassified (UNCLAS), but marked with a caveat and therefore not releasable. The DHS HSIP dataset that Andrew mentioned has parts that are unclassified, but I believe it all is marked FOUO and some segments may have additional caveats. *I do not believe it can be made available for commercial use.*

Many governments have such systems for limiting access to data. Not all are compatible, even between allies. Even for units that are in joint combat operations. And yes, that is not entirely rational. Ask a solider.

Also, to my knowledge, the World Bank GFDRR OpenDRI data do not cover any US data.

Hope this helps,

- John

Andrew Turner wrote:

Probably best would be to setup the pattern where data could be slotted in for a specific event. Kind of like a surge protector that you plug into the wall, then the data all can match any of N interfaces to provide: Infrastructure, People, Evacuation, Centers. That way it can adapt to whatever the most trustworthy/recent data that exist for that event.

Take for example my experience sitting at FEMA during Sandy. Within a few hours FEMA went through 3 different sources for 'gas availability' - from the HSIP official, to the daily update based on credit-card data, to the IMSOCIO updating feeds. This would have been impossible to identify before-hand.

#flexibility

Andrew

On Tue, Mar 5, 2013 at 1:02 PM, Patrick Meier (iRevolution) > wrote:

Thanks very much, Andrew and Simcha!

I love the idea of a Pinterest for open (disaster) data. Anyone know if the OpenDRI data is readily accessible?

In terms of what we (QCRI & MIT/CSAIL) are trying to achieve: we're looking for linked data to pull into this free & open source DIY disaster response app. This project is still at the very early stages.

Thanks again for your replies and for any additional guidance you may be able to provide.

My very best,

Patrick

On Tue, Mar 5, 2013 at 12:50 PM, Andrew Turner > wrote: As Simcha said there is no one-stop-shop nor authoritative source and there never will be. The solution is data federation. There will be many sites and services for open-data that relate to disaster response, both directly and indirectly, that are owned, managed, and provided from many agencies, organizations and companies.

In the US there is the 'official' HSIP (Homeland Security Infrastructure Program) that has all of the national infrastructure (roads, bridges et al) and is an aggregation of mostly commercial and some government sources - but then you also have OpenStreetMap. Sites like GeoCommons, Socrata, ArcGIS Online, CKAN et al. are all hosting different official and unofficial data sources. 'BigML' has a list of lists of thousands of data repositories [1]

There have also been a dozen spreadsheets and google docs to collate these. Many are event-based and short-lived and too easily forgotten about or locked away behind an opaque URL.

I think a better approach is to provide loosely coupled ways to bookmark and share these repositories. Imagine if we all use Pinboard or Delicious to tag repositories and categories that we can subscribe via RSS or API's to pull together.

I'm also hopeful that we can use lightweight data standards to promote federated search and data sharing. For example GeoCommons provides OpenSearch + Atom feeds that it uses itself to search remote repositories and pull in data. Anyone can support these particular standards, or others like CSW, DCAT or more. Again, more options that you can simply collate together as a simple exercise [2]

This is a valuable and important problem - but let's not try to crown a single repository or create yet another spreadsheet. A good discussion on what you're trying to achieve, and ways that work in the broad community as well as for responders and agencies is far more enlightening so that we can evolve a real solution.

Andrew

[1] BigML Repository list: http://blog.bigml.com/2013/02/28/data-data-data-thousands-of-public-data-sources/
[2] CivicCommons Data Standards: http://wiki.civiccommons.org/Data_Standards

On Tue, Mar 5, 2013 at 12:08 PM, Simcha Levental > wrote:

I don't think their is one authoritative source.

It's time that we embark on a effort to creating a resources spreadsheet that can assist once data is needed. This also might help us recognize data gaps that might be addressed through a data philanthropy effort.

I would be happy to lead this effort.

On Mar 5, 2013 10:25 AM, 'Susannah Dyen' > wrote:

I would also be interested.

Best,

Susannah

On Tue, Mar 5, 2013 at 9:47 AM, Kerrie Aman Carfagno > wrote:

I would also be interested in open data sets.

Kerrie