[Version 0.0.9] wadget - A crossplatform /idgames CLI client

Hey guys! I used to post here a while back. Like, years ago with a bunch of shitty commentary and projects. Now I mostly lurk /idgames and jump on the occasional Zandronum server.

Recently, I became a software engineer and started developing a project in my free time called "wadget". For those familiar with CLI applications, the goal of it is to be a "package manager" of sorts for addons in the /idgames archive. The program is written using Python3 and the /idgames API. You can check it out here: https://github.com/averyre/wadget

Basically, I want to make access to /idgames similar to synaptic or snap. You could easily "wadget doom19s" to get the Doom shareware in your working directory, for example, as explained in further detail on GitHub. There is also a search function to scrape for WAD IDs, making things a little easier for now.

The website is here: https://averyre.github.io/wadget/

Obviously, this project is targeted very specifically, but I'd love some feedback on the very beginnings I have so far!

Avery

Hey, this looks great! I would especially like it if the program included a helper for quickly and conveniently uploading a wad and readme to /incoming.

Thanks! I'm sure I could implement some sort of packaging and uploading solution shortly once I'm satisfied with the basic functionality for retrieving and unpacking WADs.

Hey guys! Sorry to double post. I've just completed the first "release" (ish) of wadget. You can install it with pip by using:

$ pip3 install wadget

A good starting point would be to try the search function, to look for all 32in24 files (shameless, because I've participated):

$ wadget 32in24 -s

Now let's download one of the results:

$ wadget 15960

Let me know what you guys think! It would be super appreciated!

wow friend!! that's really cool. i don't really know what's it's supposed to be used for, but i like it nonetheless c:

here are some pictures of it in action:

thank you for this great gift friend

4 hours ago, bonnie said:

wow friend!! that's really cool. i don't really know what's it's supposed to be used for, but i like it nonetheless c:

 

here are some pictures of it in action:

  Reveal hidden contents

 

thank you for this great gift friend

Hey, thanks a ton man! I loved being able to see the screenshots, especially within Windows as I'm on Ubuntu.

I definitely should have cleared up its purpose a bit more! So for example, I use a single folder to store my WADs (Documents/WADs) and am constantly downloading from /idgames via the website, extracting and moving my WADs to that directory. Starting a Doom game for me usually looks something like this:

$ gzdoom -file bd*

So I wanted a command-line alternative to accessing /idgames without ever leaving that WAD directory. wadget pretty much is a solution to quickly add WADs and other files via the command-line to wherever you keep your WADs! It's not very useful for, admittedly, those who use GUI's.

Sounds pretty cool! I sure could use such a thing as I already use CLI to launch prboom-plus on all platforms, anyway.

I'm gonna try it out tomorrow, probably on Windows and macOS, because my linux machine is not fully set up atm.

For now, from the top of my head, some features that would be nice to have:

1. Put WADs into the directory set in DOOMWADDIR environment variable. That's would allow WADs to be automatically ready to use with all source ports that support the environment variable (I know prboom-plus and GZDoom do, because I use them this way), without the need to store multiple copies. It would also allow putting the WADs in the right place independently of current working directory.

2. Set a storage directory for downloaded WAD archives (probably via config or something like that). My workflow with the WADs looks pretty similar to yours, although I also keep the zipped WADs in a separate directory, just in case I mess up the WADs somehow.

3. Downloading WADs by archive name rather that by id. Say, if I wanted to install Community Chest 3, I would like to type something like this:

wadget install cchest3

This one would probably require the most work, and I don't even know if the archive names are all unique, but it seems like a nice thing to be able to do.

Thanks for your input and testing! To address all your recommendations, which I love by the way!

1. I was not familiar with DOOMWADDIR while coding this, thanks for bringing it up! I will definitely write an implementation/flag of some sort to install WADs directly here, which will likely have to follow another function I'm writing to extract and grab just the WADs/pk3s/whatever from the Zip instead of keeping the archive.

2. A config file is in the works! This will have the exact feature you've requested and is on my roadmap. :)

3. Another feature I want to implement! To get around multiple files having the same or similar names, I'm thinking that it will grab the first result that is returned from a standard search, and if a specific file must be specified that can be done using the file ID as normal.

Many thanks!

I played around with wadget a bit, and works just fine on macOS 10.14 and Windows 7. I also like the way search results are presented.

Minor issues I ran into:

1. I couldn't find Alien Vendetta with it. Searches for "av", "alien vendetta", and "vendetta" yield 0 results, and search for "alien" - 21 results, but AV is not among them. It can be found via search on idgames.

UPD: I just managed to find it with wadget by searching for "av.zip".

2. Some search results have "None" where the WAD name should be, for example ID 2367 and 1036 from my "alien" search results.

3. wadget doesn't check if the file to be downloaded already exists in the directory and silently overwrites it.

Also, two more suggestions: add a progress bar for downloading process (apt-get style), and downloading multiple WADs by giving wadget multiple IDs would be nice:

wadget 12021 13024 15156

Anyway, I very much like where this is going, and hope to see more features implemented soon. I think wadget would be a great way to get that one WAD you forgot to copy to your laptop on the go, or to set up Doom on a fresh OS when you already know which WADs you want.

I think implementing the /idgames database with "magic numbers" is probably the worst thing I have ever done.

8 hours ago, Rathori said:

1. I couldn't find Alien Vendetta with it. Searches for "av", "alien vendetta", and "vendetta" yield 0 results, and search for "alien" - 21 results, but AV is not among them. It can be found via search on idgames.

UPD: I just managed to find it with wadget by searching for "av.zip".

I believe this error is because the API will not return any results if the search query is less than 3 characters (in "av"s case), but is also searching for a filename specifically. So although av *should* return av.zip, it doesn't. I'll have to write a warning for searches, or automatically append ".zip" when required. Also flags to search other fields should help.

8 hours ago, Rathori said:

2. Some search results have "None" where the WAD name should be, for example ID 2367 and 1036 from my "alien" search results.

Thanks for catching this! So when the API returns file information, it returns some information from the text file included with it. I believe, actually, the file id and filename are the only two fields it doesn't return from the text file. What I thiiiiiink is happening here, is when there is nothing valid in the text file for a name or it's a weird text file in general, the API returns "None". A workaround I'm thinking of is to display the filename directly if "None" is returned.

8 hours ago, Rathori said:

3. wadget doesn't check if the file to be downloaded already exists in the directory and silently overwrites it.

My bad! I will fix this right away.

As to your recommendations, I like them both! Those features will definitely be implemented.

7 hours ago, Linguica said:

I think implementing the /idgames database with "magic numbers" is probably the worst thing I have ever done.

No doubt! Well they work pretty well with the API, since the alternative is to pass to it a full file path. Do they just increase incrementally with uploads or are they random in some way?

Sorry to double post again, there has been another somewhat significant release! Install it with:

$ pip3 install wadget

or upgrade your previous installation with:

$ pip3 install --upgrade wadget

Notable new features and fixes:

1. Download a file by filename. You can now "wadget cchest3" for example. This will download the first file returned from a search with the specified filename.
2. You can extract and then remove a source archive automatically when downloading with the "-e" flag.
3. Appends ".zip" to filenames less than 3 characters to prevent API errors.
4. Prompts the user before overwriting an existing download.
5. Misc other bug fixes and the like.

I'm thinking because some popular wads like "av.zip" don't return what the user would expect in the response, there will be a separate "redirection" file for popular WADs to point wadget in the right direction. This is another API limitation, but can easily be worked around.

EDIT: Hotfix 0.0.8 released, simply upgrade again if required. This fix gets rid of all 'None' responses in search.

EDIT 2: 0.0.9 released. Added DOOMWADDIR support with -wd flag and a downloading progress bar.

Any and all feedback is appreciated!

Avery

On 10/9/2018 at 9:31 AM, AveryRe said:

Added DOOMWADDIR support

Nice! I'll check it out when I have time.

Here's my thoughts on version 0.0.9:

Downloading by filename feels great, it makes the whole process feel much more streamlined than running a search first. Combined with "-e" and "-wd", this allows you to install almost any WAD with a single command, which is great.

However, installing Alien Vendetta is still non-trivial, because apparently, the first search result for either "av" or "av.zip" is Nodenav, which is what gets installed if I run either "wadget av" or "wadget av.zip". This made me think that it might be a better idea to show user the search results if there are multiple matches, rather than install the first one, because it might not be what the user wanted.

And some minor nitpicks:

- I'm not particularly experienced with running Python modules on linux, but on freshly installed Ubuntu 18.10 I couldn't just run "wadget", and had to type "python -m wadget" instead. I don't know if it's something that can be fixed on the package side, but it should probably be mentioned in the readme.

- IMO running "wadget" with no arguments should show the help message rather that print out an error. It shows the "usage" line, but explaining what the arguments actually do would be more useful than an error message and a list of single-letter arguments that you don't really know what to do with.

- Show file size units (bytes, kilobytes, or whatever) when downloading.

On 10/21/2018 at 5:29 AM, Rathori said:

I'm not particularly experienced with running Python modules on linux, but on freshly installed Ubuntu 18.10 I couldn't just run "wadget", and had to type "python -m wadget" instead. I don't know if it's something that can be fixed on the package side, but it should probably be mentioned in the readme.

I'm actually using Ubuntu so I know a base install from PyPi should work. Did you specify pip3 when installing? wadget won't function under Python 2x, so rather than 'pip install wadget' you'll need to 'pip3 install wadget' on distros like ubuntu. This is, actually, demonstrated in the readme markdown and on the index page of the website!

On 10/21/2018 at 5:29 AM, Rathori said:

- IMO running "wadget" with no arguments should show the help message rather that print out an error. It shows the "usage" line, but explaining what the arguments actually do would be more useful than an error message and a list of single-letter arguments that you don't really know what to do with.

I agree! That's definitely temporary.

On 10/21/2018 at 5:29 AM, Rathori said:

However, installing Alien Vendetta is still non-trivial, because apparently, the first search result for either "av" or "av.zip" is Nodenav, which is what gets installed if I run either "wadget av" or "wadget av.zip". This made me think that it might be a better idea to show user the search results if there are multiple matches, rather than install the first one, because it might not be what the user wanted.

I'm working on a sort of "redirection dictionary" for popular WADs that aren't intuitive to circumvent this problem.

Thanks again for the help! I'll put some more work into it and be back!

On 10/23/2018 at 3:01 PM, AveryRe said:

Did you specify pip3 when installing? wadget won't function under Python 2x, so rather than 'pip install wadget' you'll need to 'pip3 install wadget' on distros like ubuntu.

Sorry for taking so long with my reply, I have a lot going on IRL. Yep, that's exactly how I installed it. I even checked my shell history, and it has `pip3 install wadget`. In fact, I don't even have pip installed for Python 2.

But anyway, I ran `pip3 install wadget --upgrade` and that fixed it, as well as about 7 or so dependencies

On 10/23/2018 at 3:01 PM, AveryRe said:

Did you specify pip3 when installing? wadget won't function under Python 2x, so rather than 'pip install wadget' you'll need to 'pip3 install wadget' on distros like ubuntu.

Sorry for taking so long with my reply, I have a lot going on IRL. Yep, that's exactly how I installed it. I even checked my shell history, and it has `pip3 install wadget`. In fact, I don't even have pip installed for Python 2.

But anyway, I ran `pip3 install wadget --upgrade` and that fixed it, as well as about 7 or so dependencies

Oops, I didn't mean to make a quadruple post. Who knew that pressing "Ctrl+Shift+Backspace" repeatedly spams the forums with the same post.

It seems I can't delete those myself, so could a moderator please clean up my mess? Thanks.

On 10/9/2018 at 8:31 AM, AveryRe said:

Download a file by filename. You can now "wadget cchest3" for example. This will download the first file returned from a search with the specified filename.

Hm. I wonder, how hard would it be to make it Do The Right Thing, that is install if unambiguous, but ask otherwise.

Consider:

% wadget install hell
Searching...
184 results including "hell" found:
[...]
329 - hellbnd.zip - Hell Bound
[...]
12610 - hell_kp.zip - Hell Keep
[...]
17259 - hellbnd.zip - Hellbound
[...]

% wadget install hellbnd.zip
Searching...
2 results including "hellbnd.zip" found:
329 - hellbnd.zip - Hell Bound
17259 - hellbnd.zip - Hellbound

% argh
argh: not found

% wadget show 329
Title:	Hell Bound
Filename:	levels/doom/g-i/hellbnd.zip
Size:	54.45 KB
Date:	06/06/94
Author:	Megaduck
Description:	Its a huge multiplayer level.

% wadget show 17259
Title:	Hellbound
Filename:	levels/doom2/Ports/megawads/hellbnd.zip
Size:	16.66 MB
Date:	06/29/13
Author:	Zoltán Sófalvi (Z86)
Description: 32 map megawad with collections of maps made from different periods of my doom mapping hobby, along with new ones. All old maps are re-polished to make them consistent in look with the newer ones. Maps are "progressively" chained together (so the previous map's end is more or less linked to the next one's start). Difficulty is medium, sometimes hard, but not insane. Experienced players can do it in UV without any problem.

HBFM29.wad: the full version of MAP29. Load this if you think your port/computer can handle the awesomeness (or laggyness) of it. Barely any gameplay difference, though.

% wadget install 17259
Downloading: Hellbound (hellbnd.zip)...
Unpacking...
Done.

%

On 10/31/2018 at 7:10 PM, Rathori said:

Sorry for taking so long with my reply, I have a lot going on IRL. Yep, that's exactly how I installed it. I even checked my shell history, and it has `pip3 install wadget`. In fact, I don't even have pip installed for Python 2.

 

But anyway, I ran `pip3 install wadget --upgrade` and that fixed it, as well as about 7 or so dependencies

No worries! And that makes sense! I added a dependency for CLINT, which in turn has it's own, so that was probably it.

On 11/4/2018 at 2:39 PM, wrkq said:

 

Hm. I wonder, how hard would it be to make it Do The Right Thing, that is install if unambiguous, but ask otherwise.

Consider:

  Hide contents

% wadget install hell
Searching...
184 results including "hell" found:
[...]
329 - hellbnd.zip - Hell Bound
[...]
12610 - hell_kp.zip - Hell Keep
[...]
17259 - hellbnd.zip - Hellbound
[...]

% wadget install hellbnd.zip
Searching...
2 results including "hellbnd.zip" found:
329 - hellbnd.zip - Hell Bound
17259 - hellbnd.zip - Hellbound

% argh
argh: not found

% wadget show 329
Title:	Hell Bound
Filename:	levels/doom/g-i/hellbnd.zip
Size:	54.45 KB
Date:	06/06/94
Author:	Megaduck
Description:	Its a huge multiplayer level.

% wadget show 17259
Title:	Hellbound
Filename:	levels/doom2/Ports/megawads/hellbnd.zip
Size:	16.66 MB
Date:	06/29/13
Author:	Zoltán Sófalvi (Z86)
Description: 32 map megawad with collections of maps made from different periods of my doom mapping hobby, along with new ones. All old maps are re-polished to make them consistent in look with the newer ones. Maps are "progressively" chained together (so the previous map's end is more or less linked to the next one's start). Difficulty is medium, sometimes hard, but not insane. Experienced players can do it in UV without any problem.

HBFM29.wad: the full version of MAP29. Load this if you think your port/computer can handle the awesomeness (or laggyness) of it. Barely any gameplay difference, though.

% wadget install 17259
Downloading: Hellbound (hellbnd.zip)...
Unpacking...
Done.

%

 

Thanks for your examples! I like your take on the interface. I'm going to be releasing a patch very soon that helps with this issue.