{"items": [{"author": "Julia", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125106736932", "anchor": "fb-10100125106736932", "service": "fb", "text": "To be clear, the point of that last paper is that that VAM approaches can mislead, not that teachers can make their students taller?", "timestamp": "1575406379"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125106736932&reply_comment_id=10100125106886632", "anchor": "fb-10100125106736932_10100125106886632", "service": "fb", "text": "→ Julia right! The last two papers are showing that VAMs find causation where none is possible.", "timestamp": "1575406477"}, {"author": "Julia", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125106736932&reply_comment_id=10100125107630142", "anchor": "fb-10100125106736932_10100125107630142", "service": "fb", "text": "→ I mean, I still remember the rap my third grade teacher made us memorize about fruits and vegetables, but I doubt it had a lasting impact on my nutrition.", "timestamp": "1575406798"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125106736932&reply_comment_id=10100125108054292", "anchor": "fb-10100125106736932_10100125108054292", "service": "fb", "text": "→ Julia the paper shows VAMs say your current teacher causes your current height", "timestamp": "1575406972"}, {"author": "Matthew", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125106736932&reply_comment_id=10100125120264822", "anchor": "fb-10100125106736932_10100125120264822", "service": "fb", "text": "→ Eh, I think that bit of the story is questionable. The thing I dislike about it is that they use a folk model of biology (teachers can't impact student heights, but *can* impact student weights, with no citations for either claim), discover that their experiment suggests teachers do impact student heights, and conclude the experiment is showing garbage instead of that their model is wrong. [To be clear, the effect size they found was larger than the effect size I thought would be reasonable; I thought it could be up to a quarter the size but it was comparable.]
The bit of the story that makes me buy it is that when they do shrinkage, the effects on height *do* go away while the effects on educational achievement remain, suggesting that it was noise. (Similarly, when they randomly mix the student scores and assign them to teachers, they discover that teachers are 'adding value' for both height and educational achievement, in a way that suggests doing the procedure without shrinkage will generate results that are pure noise.)", "timestamp": "1575410993"}, {"author": "Harris", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125106736932&reply_comment_id=10100125136592102", "anchor": "fb-10100125106736932_10100125136592102", "service": "fb", "text": "→ https://www.nber.org/papers/w26480...
Julia abstract is here
Rebuttal from an economist friend of mine:
To be fair, basically all recent academic work on teacher VA (and afaik all teacher VA metrics in the real world) apply some form of shrinkage to VA estimates, which in this context is pretty straightforward when you have multiple years of data for each teacher, to account for precisely this problem (See my former boss\u2019 work: http://www.rajchetty.com/chettyfiles/w19423.pdf). The idea is that you can exploit that you have multiple VA estimates for each teacher to estimate the fraction of the variance of VA estimates that\u2019s due to year-to-year noise, and then \u201cshrink\u201d estimates towards the mean in a way that is proportional to the amount of noise. Shrunk teacher VA estimates don\u2019t fail placebo tests like this. Still, this is a clever paper to make the point that shrinkage is really important for VA estimates to be meaningful", "timestamp": "1575416751"}, {"author": "Sam", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125107804792", "anchor": "fb-10100125107804792", "service": "fb", "text": "This is the sort of situation in which qualitative research really matters in terms of figuring out which model is most likely. How do students who have been suspended describe the experience? How do they describe the school cultures? It's certainly implausible that the *only* relevant way in which suspensions affect school culture is suspension rate.
My understanding, from working adjacent to this field, is that high school suspensions are caused by a school culture that is 1) racist and 2) focused on controlling rather than supporting students. Students who are frequently suspended are often kids who have trouble abiding by school rules either due to disability, trauma, or just plain personality issues (if you're in a school that you perceive as arbitrarily controlling, you may stop respecting those rules). The suspensions are just the tip of the iceberg in terms of how the students are treated on a day to day basis - kids who are suspended more are also disciplined more in other ways, resulting in missed academics, poor relationship with teachers, etc. All of that is going to have an effect on people's life trajectories after school.", "timestamp": "1575406890"}, {"author": "Sam", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352", "anchor": "fb-10100125108024352", "service": "fb", "text": "Bottom line is, the suspensions do matter in and of themselves, but they're a warning sign of deeper problems and you're unlikely to change the school culture in a way that's very meaningful by *just* reducing suspensions.", "timestamp": "1575406961"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352&reply_comment_id=10100125109197002", "anchor": "fb-10100125108024352_10100125109197002", "service": "fb", "text": "→ Yes: suspension rates tell us something important about what's happening at the school, but they're probably not the main issue in themselves.
The paper ends with \"Our findings have important implications for school disciplinary and criminal justice policies. In 2014, the Obama administration issued the first national guidance on school discipline, urging schools to limit suspensions and other practices that remove students from the classroom. However, with a changing political climate and little causal evidence \u2013 in support of or against \u2013 of the impact of exclusionary discipline on students, the U.S. Department of Justice and Department of Education issued a joint statement in 2018 rescinding the Obama era guidance (ED, 2018). Our results contribute to this debate by demonstrating that exclusionary discipline practices have large negative impacts on adult crime and educational attainment\".
While we do need to figure out how to make schools better, simply limiting suspensions doesn't look like it would have anywhere near the effect the authors argue for.", "timestamp": "1575407291"}, {"author": "Sam", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352&reply_comment_id=10100125110319752", "anchor": "fb-10100125108024352_10100125110319752", "service": "fb", "text": "→ This is a bit misleading though since the Obama guidance didn't refer only to out of school suspensions and expulsions but to the full range of disciplinary responses, including trips to the principal's office and in-school suspensions. A thorough approach to disciplinary disparities would probably have a much more significant effect than one that only focused on out of school suspensions and expulsions.
https://www2.ed.gov/.../colleague-201401-title-vi.html", "timestamp": "1575407704"}, {"author": "Sam", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352&reply_comment_id=10100125110604182", "anchor": "fb-10100125108024352_10100125110604182", "service": "fb", "text": "→ The Obama guidance also included recommendations for broader school culture reforms as well, for what it's worth.", "timestamp": "1575407768"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352&reply_comment_id=10100125111547292", "anchor": "fb-10100125108024352_10100125111547292", "service": "fb", "text": "→ I'm not trying to argue that. I'm specifically saying the claim the paper makes, that a policy of lower suspensions will dramatically lower adult incarcerations, does not follow.", "timestamp": "1575408081"}, {"author": "Sam", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352&reply_comment_id=10100125116846672", "anchor": "fb-10100125108024352_10100125116846672", "service": "fb", "text": "→ Jeff Kaufman fair, but I'd say that it does lend evidence to the idea that school disciplinary climate, of which school suspension rates, are one marker, *is* a driver of adult incarceration", "timestamp": "1575409931"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352&reply_comment_id=10100125118084192", "anchor": "fb-10100125108024352_10100125118084192", "service": "fb", "text": "→ I don't think the study can tell us that. It can't distinguish between \"high levels of suspensions reflect a harmful disciplinary culture, and that culture leads to adult incarceration\" from \"high levels of suspensions reflect a different student population that for some other reason has higher adult incarceration\". I don't think the study should lead anyone to change their minds on this either way.", "timestamp": "1575410285"}, {"author": "Sam", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352&reply_comment_id=10100125118563232", "anchor": "fb-10100125108024352_10100125118563232", "service": "fb", "text": "→ Jeff Kaufman no single study could, but when you combine it with the testimony of people with lived experience it is pretty persuasive imho", "timestamp": "1575410367"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352&reply_comment_id=10100125243228402", "anchor": "fb-10100125108024352_10100125243228402", "service": "fb", "text": "→ Sam I thought I responded to this, but it seems I didn't?
I think the lived experience is doing basically all the work there. The outcome of this study is one we could see in a world where we have suspensions directly causing adult incarceration, but also in a world where suspension rates and incarceration have shared causes.
The principal movement study idea I talk about above would give a way to pull these apart some.", "timestamp": "1575459398"}, {"author": "Sam", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125108024352&reply_comment_id=10100125247130582", "anchor": "fb-10100125108024352_10100125247130582", "service": "fb", "text": "→ Jeff Kaufman the study shows that at least something systemic is going on beyond just a few people's experiences. Lived experience helps us come up with the most plausible explanation consistent with the data", "timestamp": "1575464035"}, {"author": "Peter", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125109765862", "anchor": "fb-10100125109765862", "service": "fb", "text": "What is the evidence-based, RCT-supported reason to believe that there is any causative relationship between student behavior and the issuance of suspensions?", "timestamp": "1575407433"}, {"author": "Anna", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125109765862&reply_comment_id=10100125120708932", "anchor": "fb-10100125109765862_10100125120708932", "service": "fb", "text": "→ \"...Students were randomly assigned to stand up and start swearing 10 minutes into class (disruptive group) or not (control group)...\"", "timestamp": "1575411136"}, {"author": "Anna", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125114506362", "anchor": "fb-10100125114506362", "service": "fb", "text": "\"students who attended schools with 0.38 more suspensions per student per year were 20% more likely to be jailed as adults\"
Couldn't it just be that teachers are more likely to suspend people of color and/or people of lower socioeconomic status, and police are also more likely to arrest these same populations, because racism and classism?", "timestamp": "1575408974"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125114506362&reply_comment_id=10100125115249872", "anchor": "fb-10100125114506362_10100125115249872", "service": "fb", "text": "→ They control for these, though I don't trust that much: \"We also control for race, gender, special education status, and limited-English proficiency\"
For example, you could imagine teachers being more biased against Black students with darker skin, and then later police/judges being similarly biased, and their attempt to control for race would not capture this.", "timestamp": "1575409309"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125116207952", "anchor": "fb-10100125116207952", "service": "fb", "text": "Edited the post to change \"but figure that student behavior still plays a role\" to \"but figure that student-specific effects still play a role\". The former was overly restrictive.", "timestamp": "1575409607"}, {"author": "Will", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125117131102", "anchor": "fb-10100125117131102", "service": "fb", "text": "\"Factories Schools Prisons\", a 1996 book by documentary photojournalist Michael Jacobson-Hardy", "timestamp": "1575410002"}, {"author": "Jeff Kaufman", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125117131102&reply_comment_id=10100125118443472", "anchor": "fb-10100125117131102_10100125118443472", "service": "fb", "text": "→ Can you say more about how the book is relevant?", "timestamp": "1575410333"}, {"author": "Will", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125117131102&reply_comment_id=10100125119127102", "anchor": "fb-10100125117131102_10100125119127102", "service": "fb", "text": "→ Michael took these photos in the mid-90s of factories, schools and prison in the Pioneer Valley (Western Massachusetts). He was endeavoring to show with stark blacks and white photography the connections between underfunded schools in poor neighborhoods (Holyoke), low-wage factory work, and the prisons where many who grow up in those schools and working environments wind up.", "timestamp": "1575410566"}, {"author": "Sarah", "source_link": "https://www.facebook.com/jefftk/posts/10100125104840732?comment_id=10100125117255852", "anchor": "fb-10100125117255852", "service": "fb", "text": "this is what you were telling me about when I was distracted by the multiplying numbers of entre options! yes, quite.", "timestamp": "1575410037"}, {"author": "Douglas_Knight", "source_link": "https://www.lesswrong.com/posts/y6BvzBfQMN2XiMEwq#6xHvmHdaiWWvtpa8G", "anchor": "lw-6xHvmHdaiWWvtpa8G", "service": "lw", "text": "It would be nicer if there were more randomization, but it would also be nicer if more information were extracted from the few people who are randomized. For example, I know someone who participated in an RCT of breastfeeding/formula. It was aimed at a specific (acute, adverse) infant outcome. I'm not sure it even looked at other infant metrics, but it certainly did not have long-term follow-up, not even at 5 years. Not only did the study make a big investment in persuading the subjects for such little measurement, but it is now impossible to do a better experiment, because RCTs of breastfeeding are now considered unethical because of the damage their null results do to the authors' careers. (Similarly the Swedish and Australian twin registries are the right way to do twin studies.)
On the other hand, sometimes you can't randomize and you'd like to know how well you can do correlational studies. If your employer is so enthusiastic about experiments, maybe it apply that enthusiasm to itself and do an experiment to see how well its employees can do observational analysis?", "timestamp": 1575479269}]}