Force Sequential Output with SCP?

November 9th, 2024
tech
In my bioinformatics work I often stream files between linux hosts and Amazon S3. This could look like:

$ scp host:/path/to/file /dev/stdout | \
    aws s3 cp - s3://bucket/path/to/file

This recently stopped working after upgrading:

ftruncate "/dev/stdout": Invalid argument
Couldn't write to "/dev/stdout": Illegal seek

I think I figured out why this is happening:

With scp I can give the -O flag:

Use the legacy SCP protocol for file transfers instead of the SFTP protocol. Forcing the use of the SCP protocol may be necessary for servers that do not implement SFTP, for backwards-compatibility for particular filename wildcard patterns and for expanding paths with a '~' prefix for older SFTP servers.

This does work, but it doesn't seem ideal: probably servers will drop support for the SCP protocol at some point? I've filed a bug with OpenSSH.


[1] "man scp" gives me: "Since OpenSSH 8.8 (8.7 in Red Hat/Fedora builds), scp has used the SFTP protocol for transfers by default."

Comment via: facebook, lesswrong, mastodon, substack

Recent posts on blogs I like:

It isn’t always evil to ally with Steve Bannon

Scott Alexander wrote:

via Thing of Things October 19, 2026

Was the HuggingFace hacking incident scheming?

In July 2026, internal OpenAI agents hacked HuggingFace (HF) in order to figure out how their evaluation grader works. This involved a thousand agents coordinating on an improvised message board to carry out an elaborate sequence of exploits. A later grou…

via Victoria Krakovna October 18, 2026

There's no point at which turning your brain off will work

In early 2025, I started seeing people turn off their brain as they use LLMs1. They would have an LLM take an action (summarize text, write some code, etc.), and just assume that it worked2. This generally didn't work in early 2025 and the result was …

via Posts on September 18, 2026

more     (via openring)