[Geowanking] Fwd: war tweets
Dav Yaginuma
dav.yaginuma at gmail.com
Wed Sep 10 10:13:03 PDT 2008
Nice, maybe Americans (hey wait, you're a Canadian!) need a bit more ambient
awareness of our tax-paid death toll.
Could you make a version that digests a once a day report though? I'd like
to see yesterday's total and a running total once per day via device updat
On Wed, Sep 10, 2008 at 9:09 AM, Anselm Hook <anselm at gmail.com> wrote:
> Paige and I threw this together yesterday morning when we had a free hour -
> it attempts to twitter new civilian casualties in iraq.
>
> Perhaps folks can suggest ways to improve it - it is kind of brute force -
> mapping the deaths could also be done - but I'm assuming third party
> services already do that and the location is published with the twitter
> albeit using David Troy's notation.
>
> - me
>
> ---------- Forwarded message ----------
> From: paige saez <paigesaez at gmail.com>
> Date: Tue, Sep 9, 2008 at 1:33 PM
> Subject: war tweets-
> To: "makerlab at googlegroups.com" <makerlab at googlegroups.com>
>
>
> http://twitter.com/IRAQDEATHS
>
>
> #!/usr/local/bin/ruby
>
> #
> # we want to fetch posts from the iraq death count and show them
> # and we want to only check every day or so ( an external cron job )
> # basically then we take a look at the most recent 50 only...
> # and we twitter up to 50 of them basically...
> # we reverse the twitter order so that the most recent ones are first
> #
>
> twittercap = 50 # twitter this many posts max
>
> #
> # 1) fetch the data
> #
> # http://www.iraqbodycount.org/database/download/ibc-individuals
>
> require 'net/http'
>
> url = "http://www.iraqbodycount.org/database/download/ibc-individuals"
>
> data = Net::HTTP.get_response(URI.parse(url)).body
>
>
>
> #
> # 2) chew on the data a bit - masticate it good
> #
> # our first goal is to read the rather dorky csv file iraq war deaths
> provides
> # we have to skip past the header crap... so lets make a copy of the data
> without that header
> # we will just loop through all the data and pull out the good stuff and
> store it into another array
> # clearly not the "perfectly clean" way to do it but good enough and we
> don't really care
> #
> # http://www.rubytips.org/2008/01/06/csv-processing-in-ruby/
> # http://fastercsv.rubyforge.org/classes/FasterCSV.html
>
> require 'rubygems'
> require "fastercsv"
> @results = FasterCSV.parse(data) # unused approach ->
> read("ibc-individuals")
> @deaths = []
> inside_stupid_header = true
> @results.each do |death|
> if death[0] == "IBC code"
> inside_stupid_header = false
> elsif inside_stupid_header == false
> @deaths << death
> end
> end
>
> #
> # 3) put the data in our storage area... keep a copy in our belly
> #
> # http://sqlite-ruby.rubyforge.org/sqlite3/faq.html
> #
> # our second goal is to store this in a structured way so we can process it
> # i will bother to keep it in a database although i could just hold it
> memory
> # we are going to want to avoid storing duplicates - because we will call
> this multiple times
> #
> # this gets slightly more involved
> #
> # 3a) let us make a table to store the data - if it already exists this
> code will crash but thats ok
> # we wrap the whole thing in a begin / rescue and just ignore the crash if
> it happens
> #
> # The format seems to be here in the ibc file : "IBC code","Name or
> Identifying Details","Age","Sex","Marital Status","Parental
> Status","Earliest Date","Latest Date","Location"
> #
> # 3b) um, store everything...
>
>
> =begin
> require 'rubygems'
> require 'sqlite'
> # this was a manual database approach - tedious!
> db = SQLite::Database.new( "endiraqwar.db" )
> begin
> result = db.execute("CREATE TABLE deaths(id INTEGER, code VARCHAR(80),
> name VARCHAR(255), age INTEGER, sex VARCHAR(32), marital VARCHAR(64),
> parental VARCHAR(64), earliest DATE, latest DATE, location VARCHAR(255))");
> rescue
> end
> @deaths.each do |death|
> result = db.execute("INSERT INTO ")
> end
> =end
>
> #
> # 3) ok scratch the above, lets use this:
> #
> # http://datamapper.org/why.html
> #
> # lets manage the data in a way that takes advantage of object oriented
> design
> # we will grab datamapper which lets us pretend our data is a ruby data
> object
> # (instead of treating it like radioactive waste and holding it at arms
> length)
> # that means we have to define what our data "is" for ruby...
> # also we will hop over to postgres as our back end datastore - away from
> sqlite ... just because
> #
>
> require 'rubygems'
> require 'dm-core'
>
> DataMapper.setup(:default, {
> :adapter => 'postgres',
> :database => "endiraqwar",
> :username => 'endiraqwar',
> :password => '',
> :host => 'localhost'
> })
>
> class Death
> include DataMapper::Resource
> property :id, Integer, :serial => true
> property :code, String
> property :name, Text
> property :age, Text
> property :sex, Text
> property :marital, Text
> property :parental, String
> property :earliest, DateTime
> property :latest, DateTime
> property :location, Text
> property :created_at, DateTime
> property :posted, DateTime, :default => nil
> end
>
> # we do not want to do this now
> # because it would erase our database
> # DataMapper.auto_migrate!
>
> # go ahead and store all the deaths
> @deaths.each do |death|
> if Death.first(:code => death[0] )
> # puts "We already found this death #{death[1]} #{death[0]} so not
> saving"
> next
> end
> # take a second to convert the date phrase into a machine date
> death[6] = DateTime.parse(death[6])
> death[7] = DateTime.parse(death[7])
> # go ahead and make a blobby thing to hold all of this
> record = Death.new(
> :code => death[0],
> :name => death[1],
> :age => death[2],
> :sex => death[3],
> :marital => death[4],
> :parental => death[5],
> :earliest => death[6],
> :latest => death[7],
> :location => death[8]
> )
> puts "recording the passing of #{record.name} at #{record.earliest} and
> #{record.code}"
> record.save
> end
>
> #
> # 4) ok cool, now we have the data - lets twitter!
> #
> # http://twitter.rubyforge.org/
> # http://github.com/jnunemaker/twitter/tree/master/examples/posting.rb
> #
>
> require 'twitter'
> twitter = Twitter::Base.new("endiraqwar","")
> @deaths = Death.all(:order => [:earliest.desc], :limit => twittercap)
> @copyofdeaths = []
> @deaths.each do |death|
> @copyofdeaths << death
> end
>
> @copyofdeaths.reverse.each do |death|
> # publish deaths that are new
> next if death.posted != nil
> result = twitter.post("#{death.name}, #{death.age}, #{death.sex},
> #{death.marital}, #{death.parental} killed on #{death.earliest.strftime("%d
> %b %Y")} at L:#{death.location}")
> # remember that we already published this death
> death.posted = DateTime.now
> death.save
> puts "posted the death of #{death.name} #{death.code}"
> end
>
>
>
> --
> Paige Saez
>
>
> 971.227.4384
> Art.Media.Love.
>
> paigesaez.org
> makerlab.com
> red76.com
>
>
>
> --
> anselm 415 215 4856 http://hook.org http://makerlab.com http://meedan.net
>
> _______________________________________________
> Geowanking mailing list
> Geowanking at lists.burri.to
> http://lists.burri.to/mailman/listinfo/geowanking
>
--
Dav Yaginuma
http://AkuAku.org/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://geowanking.org/pipermail/geowanking_geowanking.org/attachments/20080910/504465f2/attachment-0003.html>
More information about the Geowanking
mailing list