Force Sequential Output with SCP?

November 9th, 2024
tech
In my bioinformatics work I often stream files between linux hosts and Amazon S3. This could look like:

$ scp host:/path/to/file /dev/stdout | \
    aws s3 cp - s3://bucket/path/to/file

This recently stopped working after upgrading:

ftruncate "/dev/stdout": Invalid argument
Couldn't write to "/dev/stdout": Illegal seek

I think I figured out why this is happening:

With scp I can give the -O flag:

Use the legacy SCP protocol for file transfers instead of the SFTP protocol. Forcing the use of the SCP protocol may be necessary for servers that do not implement SFTP, for backwards-compatibility for particular filename wildcard patterns and for expanding paths with a '~' prefix for older SFTP servers.

This does work, but it doesn't seem ideal: probably servers will drop support for the SCP protocol at some point? I've filed a bug with OpenSSH.


[1] "man scp" gives me: "Since OpenSSH 8.8 (8.7 in Red Hat/Fedora builds), scp has used the SFTP protocol for transfers by default."

Comment via: facebook, lesswrong, mastodon, substack

Recent posts on blogs I like:

American effective altruists should probably donate to political candidates

My recommendation for donors who are American citizens or permanent residents is that they donate directly to high-value political candidates.

via Thing of Things December 28, 2025

Opinionated takes on parenting

This post is a collection of parenting takes that sometimes go through my head, based on my experience raising our two boys (5 and 2 years old). All of this is based on my experience and might not apply to others (see the law of equal and opposite advice)…

via Victoria Krakovna December 16, 2025

How to Make a Christmas Wreath

Yesterday, I made a Christmas wreath. Here's how to make one. First, find an evergreen tree near your house. Clip off a few branches from the tree. Try to have as many leaves or needles on the branches as possible. Next, bring them home. What I usu…

via Anna Wise's Blog Posts December 6, 2025

more     (via openring)