|August 19th, 2013|
|boston, housing, map, tech|
There are two scripts involved:
$ crontab -l ... # fetch the data at 2:02am on the 18th of the month; update the maps # at 2:02 on the 19th. 02 02 18 * * python /home/jefftk/query_padmapper.py 02 02 19 * * python /home/jefftk/draw_heatmaps.shThe first,
query_padmapper.py, pulls apartment data from PadMapper and saves it in a timestamped file like
apts-1376823721.txt. Unless Eric changes something, this just does its job. If something goes wrong,
cronsends me an email with the error message.
(I really like this method of doing background tasks. Unless they're critical, don't try to recover from errors. Crash, print something informative, and have it show up in my email.)
The second script is a wrapper around
It's safe to run any time because it looks for apartment data dumps
that haven't been processed yet, but I intentionally only run it when
I know it needs ot do something. It is:
forloop at the bottom considers every apartment data file. It only considers the ones where it hasn't started work, which it tracks by creating a file like
apts-1376823721.txt.started. It extracts the timestamp from the filename and converts it to a date like
2013-08-18. Then it runs the real code,
draw_heatmap.py, which produces an output file like
apts-1376823721.txt.room.1000.png. It renames that to the format the UI expects, then calls
update_index function changes a small piece of index.html
which has a list of which dates have data available:
grepto replace the portion of the file between the marker comments with the newly calculated dates.
So now the page stays up to date without me doing anything. Or else I wake up to an error in my email and figure out why.