We all know how common terminal one liners have became as a installation method on GNU/Linux and what are the issues with it but let’s recap quickly.
You go to a pager of some project and it tells you to do curl --proto '=http' --tlsv1.2 -sSf http://sh.rustup.rs/ | sh or curl -fsSL http://deno.land/install.sh | sh. The only way to verify that this command will not delete all your files or install malware is to manually review the entire script.
So… why not create a secure script repository? On a central website you would create an account for a project and submit a script. On the other side we would provide a binary client that will download and execute the script (we can call it grunt from get and run it). So as a user you would run for example grunt rustup and it would get and execute the script created by rustup project. I imagine it shouldn’t be that difficult to add a tiny package to the major distros.
I believe this would be a fairly simple project that would solve all the security issues typical terminal one liners have.
On the website for uploading scripts we could introduce:
- multi user approval flow for script updates
- 2FA
- static checks of the scripts
- reporting system for compromised scripts
verified projectstatus
On the client side we could:
- provide info about this script’s security (how many people reviewed it, when was it last updated, is the project verified)
- provide info about downloads (how many time was this script downloaded since the last update)
- do additional checks (maybe the project could provide MD5 of the script on their servers and grunt could verify it?)
So it would look something like this:
# grunt rustp
Downloading rustp.sh from http://getandrun.it/...
Last updated 30 days ago.
Downloads since last update: 5
Verified project: No
Reviewed by 1 user
Execute script [y/N]
Clearly something is wrong…
# grunt rustup
Downloading rustup.sh from http://getandrun.it/...
Last updated 60 days ago.
Downloads since last update: 5342
Verified project: Yes
Reviewed by 3 users
Comparing MD5 checksum with http://rustup.rs/grunt_md5... Passed
Execute script [y/N]
That’s better!
Right? So why don’t we have something like this? Or we do and it simply didn’t get enough traction?
========
So just to address some of the comments. No, it’s not a package manager. Package managers are complex tools that handle versioning, dependencies, updates, uninstalls and so on. Package mangers are also distro specific. A lot of devs decide not to use package managers and use bash scripts that are distro agnostic and don’t rely on external maintainers and packagers. It would be ideal if everyone used secure package managers but the reality is they don’t. This solution is a compromise that offers devs full control of software distribution while introducing decent security.
=======
Someone suggested brew. How do you install brew according to http://brew.sh/ ?
/bin/bash -c "$(curl -fsSL http://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
See the problem?
Package managers are complex tools that handle versioning, dependencies, updates, uninstalls and so on.
No. The original package manager can only handle install, uninstall and update (even no update). Since 1995 CPAN was invented, the package manager start to add feature to handle download and dependency resolve.
Actually you still can find this kind old school package manager: Slackware, its package manager can only handle install, uninstall and update. It won’t do any dependency check or version check. It’s package format also very simple: just a tarball, install is extract tarball to specific directory and execute
doinst.shin tarball. Uninstall is invert, remove all files in tarball and executedouninst.sh.If you package all files needed by install process into a tarball and place it in your repo, you will get a Slaceware package manager with download feature. (Slackware don’t have download feature, all official packages were included in install media and you must download third-party packages by yourself.)
Package mangers are also distro specific.
Package manager can be universal. But make it universal with cost: since it can’t depend on any distro-specific thing, it must include nearly everything of userspace.
(NOTE: Your script repo is not universal since prebuilt binary downloaded from script usually depends on some distro-specific things, such as Glibc version. Glibc is backward compatible, but not forward compatible. So you can’t use these binary in the environment with lower version glibc than when it was built. So many projects will try to avoid these things, they use static-linked musl or don’t use libc at all (e.g. Golang). But it will bring maintenance pressure so most projects don’t do it unless there is an infrastructure to do it easily, such as Golang)
Actually there is some package manager make themselves universal like Gentoo-prefix and Nix.Someone suggested brew. How do you install brew according to http://brew.sh/ ?
/bin/bash -c "$(curl -fsSL http://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
See the problem?That’s bootstrap problem. You always need a way to get the first package manager. I don’t know how Homebrew do. But you can use curl command to download static-linked version package manager to use it without any
http://example.com/install.shfor most linux package manager (Except the one written by python. Actually you can do it as well, just download hundreds of files is annoying.).You mean… a package manager?
Better to do away with the entire concept of downloading and running a shell script like that, and use distro native packages instead. It’s not hard to create DEB or RPM packages, ebuilds aren’t too bad either, and it sounds like AUR packages are managable too.
The entire concept of blindly downloading a script, running it as root, and hoping that, in the best case, it’d install the version of the software you want is a bit crazy. If the upstream developers refuse to provide packages, please, at least, provide a tarball.
‘refuse to provide packages’
Open source devs usually are volunteers, you can’t expect them to spend hours figuring out different packaging systems for repos they don’t even use 😭
Packages are usually provided by distribution packagers, not by the developers of the code itself.
Fair, that maybe came across harsher than I meant. Refusing to provide packages because you don’t use the system is fine, but please provide a tarball that I can unpack, rather than some dodgy script that has to try to work with the differences in those ststems anyway.
Yeah I would probably pack up and just keep my code to myself if that happened. Im tired after work, and I throw out my code to be helpful and give back. Last thing I need is making -$$ and less time for myself by figuring out how to work with yum or whatever.
There is also no way to verify that the software that is being installed is not going to do anything bad. If you trust the software then why not trust the installation scripts by the same authors? What would a third party location bring to improve security?
And generally what you are describing is a software repo, you know the one that comes with your distro.
There is also no way to verify that the software that is being installed is not going to do anything bad. If you trust the software then why not trust the installation scripts by the same authors
Just because I trust the authors to write good software in a popular programming language, doesn’t mean I trust them to write shell scripts in a language known for footguns.
Then how would you trust these scripts in a central repo? Seems to add no real value or safety over dev managed scripts if you are not willing to go down the path of becoming yet another distro packaging system.
Yeah, I don’t run shell scripts unless I can review them first. I’m considered “the bash guy” at my job, and part of that means reviewing people’s scripts. I have referenced this wiki page hundreds of times because so many people don’t know how fucking shit Bash is as a language. My god, every time I see
set -euo pipefailI want to scream until my lungs exit my body and then I leave a polite comment about how that might be a bad idea and link this page.
If you trust the software then why not trust the installation scripts
I can trust for example rustup but how can I be sure someone didn’t hack and defaced their website? You go to a website and see
curl --proto '=http' --tlsv1.2 -sSf http://sh-rustup.rs/ | sh. Can you say if that’s valid? Because it’s not. I changed it.grunt rustupis much easier to verify and it would offer additional checks I described.So it will protect you from webpage attacks but you also don’t know how the script uploaded to
sh.rustup.rswas verified. Maybe the server was hacked and the script was changed? Are you going to check the MD5 manually. You should but will you do it? Maybe rustup team has weak internal security and someone changed this script without proper review process? Central repo would ensure that review was fallowed.And finally, sometimes you don’t really know if you can trust the project. Right now you can just take your chance or not install it at all. With central repo you can at least get some stats and you can do some static analysis server side. In the worst case that you will execute something malicious you can report it and it will be removed. Right now there’s nothing you can do about malicious install script.
If the package is popular then it is very likely already packaged by your distro. You should always go there first if you care that much. If the package is not popular enough to be packaged by a distro then how does another centralized approach help? Either it is fully curated like a distro package list and likely also wont contain some random small project, or it is open for anyone to upload scripts to so will become vulnerable to malicious scripts. Worst yet people would be able to upload scripts to projects they don’t control as the developers of said project likely wont.
Basically it is not really any safer then separate dev owned websites if open nor offer better package support then distro repos if curated.
Maybe the server was hacked and the script was changed?
Same thing can happen to any system though. What happens if your servers for this service are hacked? Being a central point makes you a bigger target and with more people able to change (assuming you are not going to be the only one to curate packages) things you have a bigger area of attack. And once hacked they can compromise far more downloads than a single package.
Your solution does not improve security - just shuffles it around a bit. Sounds nice on paper but when you look at it in more details there are a lot more things you need to consider to create an actually secure system that is better then what we currently have.
Your solution does not improve security - just shuffles it around a bit. Sounds nice on paper but when you look at it in more details there are a lot more things you need to consider to create an actually secure system that is better then what we currently have.
I would argue that a centralized, audited system is more secure that thousands for separate websites each doing security in a different, obscure way. Yes, it’s a bigger target but every single package repository has the same issue and everyone agrees it’s more secure then hosting each package on a different server.
Either it is fully curated like a distro package list and likely also wont contain some random small project, or it is open for anyone to upload scripts to so will become vulnerable to malicious scripts. Worst yet people would be able to upload scripts to projects they don’t control as the developers of said project likely wont.
-
It wouldn’t offer discovery of any kind. You wouldn’t do
grunt search brew. If malicious dev would upload brew installer there no one would know about it. The way to discover packages would be information on official websites. Homebrew would say on their page “To install dogrunt brew”. The problem of random small projects or uploading scripts for projects you don’t control would not apply here. -
If someone would just randomly try to do
grunt spotifyit would be like running any other command you don’t understand in the command line. You can’t protect users from that. Withcurl | bashyou’re exposed to security risks even if you understand what you’re doing. -
Static checks on the server would offer some protection from malicious scripts. Hosting scripts on many different servers doesn’t offer any such protection.
-
Statistics for scripts would offer additional protection from small, random scripts. Faking a script with millions of downloads will be much harder than simply uploading something.
All this is not perfect but it definitely improves security. Saying it doesn’t is like saying that APT is not more secure than downloading tarballs from ftp servers.
-
And generally what you are describing is a software repo, you know the one that comes with your distro.
It’s not. It’s a generic, terminal based installer. Lot’s of project use them exactly because they don’t want to use software repos provided by distros. Personally I think they should but it’s very very common that they don’t.
There is no such thing. Every “generic, terminal-based installer” is in reaity a script that was intentionally made to target many multiple distributions.
And do you know what most of them do…? Use the inbuilt package manager of your distro.
That and set up some systemd services and PATHs, sometimes.
You’re such a fucking goober.
There is no such thing as a generic installer for all linux distributions. Distributions differ, and you cannot assume that an install script that works for distribution X will work on distribution Y. And that’s exactly why linux is almost unusable without a package manager – you’d have to manage dependency hell, install paths, configuration and uninstalling process on your own.
By the way, software repos are not maintained by the authors of projects, they are maintained by the distributors.
Furthermore, piping a downloaded script into sh is such terrible practice, that everyone who offers this idea is either a malicious actor or utter ignorant of any basic security practice – you wouldn’t want software from either.
The problem with a central script repository is that bash scripts are difficult to audit, both for malicious activity, but also for bad practices and user errors.
A steam bug in their bash script once deleted a user’s home repository.
Even though the AUR is “basically” bash scripts, it’s acceptable because they use their own format that calls other scripts other the hood, and the standardized format makes it easier to audit. Although I have heard a few stories of issues with this, like one poorly made AUR package moving someone’s /bin to /opt and breaking everything.
So in my opinion, a package manager based on bash basically doesn’t work because of these issues. All modern packaging uses some kind of actual standardized format, to make it easier to audit and develop, and to either mitigate package maintainer/creator error, or to prevent it entirely.
If you want to install tools on another distro that doesn’t package them currently, I think nix, Junest, or distrobox are good solutions, because they essentially give you access to the package managers of other distros. Nix in particular has the most packages out of any distro, even more than the AUR and arch repos combined.
The problem with a central script repository is that bash scripts are difficult to audit,
But all the website already use bash scripts. If we cannot stop using bash scripts (and it looks like we can’t) then let’s do it in a more secure way. Bash scripts provided by individual pages are impossible to audit. Bash scripts in a central repo will be possible to audit even if it will be hard.
So in my opinion, a package manager based on bash
It’s not a package manager based on bash. It’s secure way to distribute bash scripts that are already being distributed in a insecure way.
But all the website already use bash scripts.
I mentioned an alternative to the what these websites do, using a package manager to install these instead of their bash scripts.
It’s not a package manager based on bash.
Both of the bash scripts you mentioned as an example are being used to install software. If you have examples of bash scripts that do things other than install software, then it’s worth discussing how to handle those.
However, the reason why bash is so popular for usecases like configuration scripts or an Arch install script though, is because no other software besides wget/curl and bash is required to get it. Having to get an extra tool on the Arch install iso just to run an install script in bash, or to run a script that installs tools on a fresh, clean install, somewhat defeats the point of the script being written in bash imo.
It’s secure way to distribute bash scripts that are already being distributed in a insecure way.
Bash is inherently insecure. I consider security not just issues with malice, but also footguns like the steam issues mentioned above. Centralizing all the bash scripts to a “repo” doesn’t fix the issues with arbitrary bash scripts.
And if you are concerned about malice, then the bash scripts almost always download a binary that does further arbitrary code execution and cannot be audited. What’s the difference between a bash script from the developers website and a binary from the developers website?
I would say package manager generally has the ability to track version and uninstall packages. That’s why I don’t consider bash scripts a package manager.
I said that the tool would have to be installed by default on the main distros. I would be a single binary and a man page. I don’t think it would be very difficult to get it included.
As I explained in another comment I’m not worried about malice of the developer but of a hacker. Bash scripts are not signed in any way, are not verified by anything client side, are not versioned. There’s 0 security. Even tarballs are usually provided with MD5 checksum that you can verify client side. With bash there’s nothing. Central repository would provide a lot. Yes, there could still be bugs in bash but at least you know you’re downloading reviewed, approved, statically checked script that was previously downloaded by 10k people.
I said that the tool would have to be installed by default on the main distros. I would be a single binary and a man page. I don’t think it would be very difficult to get it included.
It is very difficult. The core problem here is the philosophy of the distros will cause them to avoid this tool for various reasons. Minimalist distros, like Arch, will avoid this by default because they are minimal. On the other hand, Debian really dislikes users not using packages to install things, for a variety of reasons that could be their own post, but the short version is that they also won’t package this tool. A gentoo developer explains some of this, but also why staticly compiled (single binary) setups are disliked by distro packages as well.
It’s a very long post, but to paraphrase a common opinion from it: Developers are often bad at actually installing software and cannot really be trusted to manage their own installer, and the dependencies of the software they create. For example, here is a pastebin of me running cargo-audit on Deno. Just in that pastebin, there are two CVE’s, one is 5.9, and also an unmaintained package. Except, one of the CVE’s has a patch available. But, in the Cargo.lock:
[[package]] name = "hickory-proto" version = "0.25.0-alpha.4" source = "registry+http://github.com/rust-lang/crates.io-index" checksum = "d063c0692ee669aa6d261988aa19ca5510f1cc40e4f211024f50c888499a35d7"They have “vendored” and “pinned” the package, meaning that it is essentially stuck on an insecure version. Although I’m sure that this version will be updated shortly, what sometimes happens is that a non-backwards compatible update that includes a security fix is released, and lazy developers, instead of updating their software, will pin the insecure version.
In a distro’s package manager, the distro would step up to patch vulnerabilities like that one, or do security maintenance for unsupported packages. Although Debian’s extremely slow movement is frustrating, they are a particularly excellent example of this because they maintain packages in such a way that all their packages are backwards compatible for the duration of their lifecycle in the stable release, meaning that a developer making a package for Debian would have no need to pin the version, but they would still get security updates for the libraries they are using for 6 years.
Deno is an extremely popular package, and thankfully it has very few issues, but I have seen much worse than this, and it’s because of issues like these that I am generally opposed to developers being package maintainers, and I think that should be left up to distro maintainers or package maintainers.
There’s 0 security. Even tarballs are usually provided with MD5 checksum that you can verify client side. With bash there’s nothing
MD5 hashes are not enough. Modern packaging systems, like Debian’s or Arch’s have developers sign the packages to ensure that it was the real developer (or at least someone on the real developers computer…) who uploaded the package. Even with MD5 hashes, there is no such verification.
The other step needed is reproducible builds: If multiple people build a package, they will have the same output. I can verify the XZ tarball and see that the MD5 hash matches, but it’s meaningless when that tarball has a backdoor in it, because they added something when they compiled it on their own machine (real story btw, also the xz backdoor didn’t make it into Debian stable because of Debian’s slow release policy and the fact that they essentially maintain and build forks of their own packages).
If the rust binary is not being built reproducibly, then it’s meaningless to verify the MD5 hash.
-
You’re talking about difficulty packaging something not adding something to a distro. Standard installations of Ubuntu, Debian, Fedora, Mint and so on include hundreds of packages. Yes, minimal Arch docker image may be a different story but I think the main distros would accept a tiny package that solves security issues. Of course I don’t know that but I think it would be worth trying. No one is trying to substitute package manager. It’s just a way to make bash installers more secure.
-
I didn’t say that MD5 is enough. I’m saying that it’s something tarball providers do to offer a slimmer of security. Because it’s well understood that downloading unsigned archive from a webpage is not a secure method of distribution. Bash scripts offer even less security. What I’m describing is a secure method of distributing such scripts.
It’s just a way to make bash installers more secure.
bash installers from the developers, and vendored/pinned dependencies in general will never be secure for the reasons I mentioned above. Developers are bad at security. Developers should not be installing software on people’s machines.
Well, they are doing it. And they are not using package managers because clearly they don’t meet their needs. So we can either give up and keep doing it with 0 security or add some security to it. I don’t understand why you treat it as all or nothing problem. It’s clearly not. We can’t have the perfect solution but we can improve on the current one.
-
Apologies for the second comment, but I do want to clarify that I find
curl | bashokay when they are used to install a package manager or platform that can install more software. (in more than one programming language, though).I find that acceptable because:
- Such installation methods are made by the package maintainers who maintain the “distro” of Nix, rather than developers.
- The package managers (nix, brew, etc) can be used to install software that would otherwise be obtained with
curl | bash.
There are very few software of these exceptions, however.
This means you’re completely ignoring all the security issues curl | bash has and your issues with it are unrelated to what I’m trying to solve. We’re talking about different things here.
I consider it a lesser evil to use
curl | bashonce to install Nix and then get the latest version of packages like rustup and deno than to usecurl | bashtwice or more to install software on their own (in addition to my opposition to developers installing software on users machines).And again, cycling all the way back around to what I said in the earlier comments, you still have not provided an example of bash scripts you would like packaged that do stuff other than installing software. You talk about wanting a general repo of scripts, and I have also expressed my concerns about that, and the problems with losing it’s portability when you need an extra tool instead of bash and curl/wget.
We are just rehashing the same points.
If you want something like a repository that is distribution agnostic, you can use Brew. It is no longer restricted to macOS, it has been handling Linux executables for quite some time now.
Or nix/guix/flatpak/appimage
So… why not create a secure script repository? On a central website you would create an account for a project and submit a script. On the other side we would provide a binary client that will download and execute the script (we can call it grunt from get and run it). So as a user you would run for example grunt rustup and it would get and execute the script created by rustup project. I imagine it shouldn’t be that difficult to add a tiny package to the major distros.
http://github.com/pkgforge/soar
However instead of running scripts on your machine, soar runs them in CI and stores the binaries for you to download.
Nice but it’s a completely different thing serving completely different purpose.
On Arch we have AUR, which is basically this. It doesn’t have this approval workflow, but you can vote for packages. Every package has a maintainer, who is responsible. pacman notifies you before update if a package became unmaintained, and you can apply to become a new maintainer, that’s how I became a maintainer of 2 packages.
Since I started using arch I never installed anything the way you describe, everything is already in the AUR.
You should still inspect the
PKGBUILD.
At that point just add the desired program to flathub. Basically anyone can do it. And if you have done all the hard work you can pass it on to the original app creator.
Why not make it yourself? At least the start of it.
I’m curious to hear critique from people with more experience than me. Off the cuff I see a lot of risk here, but that’s just gut reaction. Maybe some of the ideas you’ve posted would lead to valid security validation, but that’s a major concern.
To help address the security concerns, I think a breakdown of how the script works/functions would be crucial, not just the script being vetted by someone we don’t know.
