EPI, the facts.

Since it has been brought up recently, I've decided to air out the facts about the "Top secret community project" [sic] known as EPI. It is also my request, and being that I am an Admin here, one that I will enforce -- that any comments about this go into a separate thread. First to start one has my blessings for collecting comments.


[color=blue]This post will be locked along with the thread; everyone shall respect this. A copy will also be retained else wheres.[/color]



[i][u][b]
Project Status?
[/b][/i][/u]

Stalled (i.e. postponed until further notice), because of "Real life" taking priority for the developer.


4 people were involved in the project at it's height, and provisions made to approach 2 or 3 others at a later date. In answer to some peoples questions, Yes Graedus and Myself were involved; the "One guy" tasked with all the coding tasks was none other then myself. I enjoyed it greatly while off-work hours permitted.



[i][u][b]
EPI, what the heck is that?
[/i][/u][/b]

EPI is short for "Encapsulated Pacakge Installer". It is a method for integrating existing FreeBSD software management systems with an easier to use means for distributing third party packages (like PBI) and the ability to integrate with any desktop environment or user interface system.


[i][u][b]
EPI Project Goals?
[/i][/u][/b]


[list=1]
[*]Remove the need for the user to deal with the "Package, Port, or PBI" question.
[*]Make creating "EPI" files a snap with a minimal of fuss on the maintainer, and trivial to automate the build (but not like PBIs build system)
[*]Allow support of console, and desktop environment agnostic installation without inconvenient to maintainers.
[*]Create something that could be owned, managed, and operated by the community as a whole; not the PC-BSD developers: who proved to be incompetent and incapable with their miss management of PBI, at least in our (4 sets of) eyes.[/list]

It was intended that once the system was developed, that it would be further refined by the community and someday replace the PBI system, becoming the [i]de facto[/i] standard way of managing software on PC-BSD. Like wise, it was also intended that once the EPI system matured, it would become the means by which PC-BSD itself would manage system software - instead mucking with peoples ports.


While EPI is not my concept of what kind of package management system PC-BSD needs, it is our concept of what PBI should have been in the first place, and what [i]PBIs[/i] could have become if properly managed by PC-BSDs developers.



[i][u][b]
How does it (EPI) work?
[/i][/u][/b]


First a FreeBSD port is created for the given program; this has been done for most software that is worth running on FreeBSD and should be done for anything else. There are is approaching 21,000 programs in the ports tree, so much of that is done for us all.


Second a maintainer writes out a description of the particulars. This basically amounts to stating:

[list]
[*]What port(s) does this EPI provide?
[*]What EPI does this EPI depend on?
[*]Who are you?
[*]Special Needs Hook[/list]


By stating what ports to be provided, for example "www/firefox35". The build system would then automate the process from there on without maintainer intervention. Firefox would be fetched, built, and stripped down to the minimal required dependencies. This would all be done inside of a private jail on a build server, where in there is nothing else to interfere with the process (that is, inside the jail).


The "Firefox EPI" would depend on several other EPI, in this case it would depend on the following ones: EPI Core Services (the EPI system tools), X Windows System, and GTK+ Runtime. Because of this issues relating to X and GTK dependencies are removed from the Firefox EPI, creating a *much* smaller download and more manageable interface for people who just want to install Firefox, and have it just freaking work without trouble! Because of this design decision, unlike with PBI; the issue of dependencies are automated and can be checked. PBI does not support that. An advantage of EPIs way of doing it, results in ease of maintenance, more effective use of disk space, and more effective integration with FreeBSD. Another great perk is it makes writing things like Flash or MPlayer plugin EPIs much less painful then with PBIs.


For security reasons the EPI file would be digitally signed during the creation process. Every maintainer has their own "Key" that is used for signing EPIs that they create. This allows a package to be traced back to its creator, who must manage their reputation within a "Web of Trust" distribution model.


In case of "Special Needs", there is a special Encapsulated Package Installation Language, or "EPIL" for short. EPIL is a simple declaratory way of scripting the installation process. This is analogous to the various 'PBI.*.sh' Bourne Shell scripts used in PBI. [b]Unlike PBI, the EPIL system is designed for use by non-programmers and is rarely required[/b]. Every thing that can be done for you, will be done for you, in order to create the best possible encapsulation of work, minimize your hardship, and make life easier on both users and maintainers. By contrast creating a PBI requires an understanding of UNIX shell scripting and programming, and employs a flawed Application Programming Interface, which usually results in poorly created PBI and fewer maintainers. EPI solves this problem by applying sound software engineering practices, and makes creating an EPI a snap. [b]Under normal conditions the maintainer has to never write an EPIL script, and even then it is less trouble then writing a forum post[/b]. The maintainer has no need to worry whether the installation is text, graphical, attended, or unattended; all standard needs are done by magic; that is the massive opposite of traditional PBIs.



After creation, the EPI file makes its way to a sacred repository for evaluation; a way of downloading it is provided to the author. Trained people inspect the maintainer serviceable parts, i.e. no hidden delete all files on your system kind of bug. Both those trained folk and regular but trusted people then test the individual EPI to make sure it works as advertised. A simple check list is used to note that down correctly, reports shall be publically posted and the maintainer notified.


If no show stoppers were found and the maintainer is in good standing with the community authority, their package is then hosted for public download in accordance whatever community policy is deemed appropriate. The community website (think like the PBI Directory) would then host a download link and all necessary data, so that end users may download the created EPI.


If enough end users complain or have problems with EPIs created by a specific maintainer, that maintainers rights to use the community systems will be temporarily revoked (permanently if need be), and the maintainers "Key" will become untrusted by the EPI Community Authority - thus invalidating our trust in that maintainers EPIs safety for general consumption; individual end users have the ability to ignore the communities decision and install those EPI anyway.



A end user then downloads the EPI file, it could be from the Community Authorities website, from a friend, or even direct from the third parties website! (E.g. Adobe, KDE, Gnome, etc.)


The end user chooses the installation method: graphical or textual.


To install via graphical mode, simply double click the .epi file on your desktop and it will begin a graphical installation wizard. The wizard run is user servicable with a default based on your environment; i.e. Gnome & Xfce users get a GTK+ based wizard, KDE users get a Qt based wizard. Users and Developers could create their own wizards.

To install via textual mode, simply run the installation program in the shell:

[code]
sample# epi-install ./Mozilla_Firefox-3.5.3-i386.epi
[/code]


Both methods invoke the same program, but with different arguments.



The epi-install program updates its understanding of "Trusted Keys" published by the Community Authority, or any other source the user chooses to trust; the user can even skip this step.


Assuming all has went well, epi-install then unpacks Firefox accordingly, verifies the maintainers signature and the packages integrity. If found, the compiled EPIL script is run - the user can choose not to run the script. Normally this is a moot point, because there shouldn't be any script needed. Of course, the EPIs installation is recorded in a database.


What the user sees depends on how it was run. In text mode they get a console friendly way of doing the installation. In graphical mode, they get a GUI install wizard like PBI. Environment variables and command line switches are provided override behaviour - for example, choosing to run the Qt wizard under Gnome. All this is so easy because the EPI maintainer was never arsed with dealing with it, it was done automatically for them.



Firefox is now installed, the end user can run it from its locatiion in $EPI_ROOT. The default $EPI_ROOT would likely be /usr/epi if adapted by PC-BSD. When "Installed as a third party product" on FreeBSD or PC-BSD, the default $EPI_ROOT would likely be /usr/local/epi.


Our way of doing things would enable both shell users and desktop users a fairly painless way of accessing firefox, without favoritism to KDE or Gnome.


[i][u][b]
Ok, so how does this relate to PBI?
[/i][/u][/b]


PBIs are managed by the PC-BSD developers, and the people trusted with watching over the safety of end-users are either corrupt, derelict, or incompetent. [i]EPI[/i], would instead be placed into community hands, so that no one person or entity has total control.


As a format, how they work is very different. An EPI is a compressed archive containing program files and meta data; an external program located on the users machine is used to handle the installation procedure. This is how many package management systems designed for UNIX work, Microsoft's own Windows Installer is not to far off either. APT, DPKG, RPM, and FreeBSD Packages work this way as well. The PBI format on the other hand, is a self extracting executable with an embedded archive containing additional meta data, program files, and an embedded installation wizard. The PBI system is dependant upon the FreeBSD version, KDE version, and the presence of system programs -- PBI is written in C++ but done like a shell script. Internally, PBI is both ugly and non-unix like. [i]EPI[/i] instead provides a more platform and version independent way of doing things.


The format of PBI files, how they work, what they do, and how they are managed by the system is generally undocumented. [i]EPI[/i] would provide a totally documented system, making it easy for developers, system administrators, end users, and businesses. Heck, you could even create your own EPI system that is totally compatible - you can't do that with PBI, unless you read a lot of bad code and kuddle up to a nasty API.


In order to make a PBI, you need to know quite a bit about shell script and do a lot of leg work that should be done for you automatically; end result is most PBI install scripts are bad, even Kris Moore's are shotty. [url=http://sas-spidey01.livejournal.com/389068.html]I found an old PBI script that I wrote a while back[/url], that is done 'properly'.


Because [i]EPI[/i] files are throughly checked at install, it tries to ensure that what you download is exactly what the maintainer created. By contrast the PBI file you download is not guaranteed to be what the maintainer created, there is no safe guard against tampering with the files on the mirror - the user is on their own without so much as a checksum of the actual .pbi file that was sent to the mirror!



[i]EPI[/i] has a simple and documented dependency model. If you don't have GTK+ Runtime EPI installed, Firefox EPI should warn you. The EPI Core Services provides as a contract, a set of ultra-common dependencies used by many programs. This reduces disk space waste and allows EPI to work across differing versions more easily. Our decisions would have created a more robust system then PBI, while minimizing dependency hell to the same level. The way PBI does things creates more work on the maintainers and cause more interoperability/integration problems that the PC-BSD developers simply don't care to address. PBI with hidden or undocumented dependencies are also not uncommon, because the only 'standard' of what the PBI can depend on is the release notes for PC-BSD X.Y.Z and the PBI "Guidlines" as they have become, which used to be rules that were just often thrown in the trash can.



[i][u][b]
OK, OK, enough already, but who the heck are you Terry?
[/i][/u][/b]


I am a person who has used computers almost since he learned how to walk. Someone that loves software engineering, and shares his grandfathers work ethics, that if it has your name on it, then it has to be GOOD.


Several years ago, I encountered UNIX and embraced it with an open heart.


The first date with PC-BSD was 1.0RC1, a release candidate shipped in 2005. I have since various versions of PC-BSD on home servers, desktops, and laptops. During the 7.0Alpha testing cycle, before KDE4 had even entered the picture, I had made the decision to transition my personal-workstation to FreeBSD, and never looked back.

As a part of this forum, I joined to see if I could help people and learn a thing or two from my superiors. Even after they left, I remained, eventually becoming a Moderator and later an Forum Administrator at the request of Kris Moore - likely because I shouted loudest of all about the spam problems. My activity on the forums over the past ~two years has rubber banded with my work schedule and the rest of living.


For a time, I created PBIs of programs that interested me, such as Blackbox, which was the first alternative to KDE to be had in PBI form. After a while, flaws in the PBI design and the developers disregard for their own rules caused me to "Give up" on creating and maintaining PBIs. I have seen everything to be seen with PBI, down even to the point of Charles breaking the PBI rules, PBI developers publishing their own PBI without testing, Kris Moore changing the rules after breaking them for Win4BSD, Kris changing them back over community outcry (and increasing lousy PBIs lol). Throughout it all, the process of getting PBIs published has made me sicker then watching a corrupt Government at work.


I've generally kept a safe distance from PC-BSD development, this is why I never became involved with the development team; their actions over the years also do not give me desire to "Volunteer" my services.



When the question that PBIs could be created automatically was brought up many moons ago, it was shot down, most strongly by none other then the (then) host of the PBI Directory website (Charles). I supported the idea and it was generally held in contempt as something "Impossible", only later to see it become a part of PC-BSD. Exploring it ahead of everyone else, was actually how I learned much about shell scripting.



My skill set includes C, C++, Java, Perl, PHP, Python, and all the way to parts of X86 assembly, Scheme, and Common Lisp. More task specific issues such as SQL, HTML, LaTeX, sh, bash, batch/cmd, AWK, SED, and indeed, even [i]ed scripting[/i], are apart of my abilities.


I think no one will debate that I know a thing or two about that which I speak.