How to Use curl in a Bash Script to Download News Articles for Offline Viewing

Chat history

Ever wish you could save your favorite news articles to read offline, whether for an airplane ride or a cozy, no-WiFi spot? Using curl with a Bash script is a great way to download web content in seconds. This guide will show you how to create a script to fetch articles with curl, then add some options for those who want to personalize the process.

 

 

Why Use curl?

curl is a command-line tool for transferring data to and from servers. It’s fast, flexible, and widely supported, which makes it perfect for downloading articles, images, and more with a simple script.

---

Step 1: Setting Up the Script

First, let’s set up our script to download a list of articles. Create a file called download_articles.sh:

nano download_articles.sh

Copy this starter script into your file:

#!/bin/bash

# Directory to save downloaded articles
SAVE_DIR="$HOME/Articles"
mkdir -p "$SAVE_DIR"  # Create the directory if it doesn't exist

# List of article URLs to download
URLS=(
  "https://example.com/news/article1"
  "https://example.com/news/article2"
  "https://example.com/news/article3"
)

This code will save articles to an Articles folder in your home directory and sets up a list of example URLs to download.

---

Step 2: Download Articles Using curl

To download each article, we’ll loop through our list of URLs and use curl to fetch each one. We’ll use basename to give each file a meaningful name based on its URL:

for URL in "${URLS[@]}"; do
  # Extract a filename from the URL
  FILENAME=$(basename "$URL").html
  
  # Download the article with curl
  curl -L "$URL" -o "$SAVE_DIR/$FILENAME"
  
  echo "Downloaded: $FILENAME"
done

Here:

• basename extracts the last part of the URL (e.g., article1), and we add .html for readability.
• curl -L follows any redirects (handy for news sites).
• -o "$SAVE_DIR/$FILENAME" saves the file in our Articles folder with a unique name.

Run the script by making it executable and then executing it:

chmod +x download_articles.sh
./download_articles.sh

Step 3: Customizing the Script for Extra Flexibility

Want to give your script a personal touch? Here are some easy modifications to make it even more versatile.

A. Prompting for URLs

You can allow users to enter URLs each time they run the script:

echo "Enter URLs to download (separate by spaces):"
read -a URLS  # Take input as an array

Now, when the script runs, it’ll prompt the user to enter article URLs, which it will then use to download.

B. Setting a Custom Save Directory

To allow users to specify a custom directory:

echo "Enter the directory to save articles (default is $SAVE_DIR):"
read CUSTOM_SAVE_DIR
SAVE_DIR="${CUSTOM_SAVE_DIR:-$SAVE_DIR}"  # Use default if no input

mkdir -p "$SAVE_DIR"  # Ensure the directory exists

Now, the user can specify a directory to save articles, or press Enter to use the default Articles folder.

C. Downloading in Different Formats (PDF)

Some websites offer PDF versions of articles. If so, you can specify these URLs to download PDFs instead of HTML:

URLS=(
  "https://example.com/news/article1.pdf"
  "https://example.com/news/article2.pdf"
)

Tip: Many sites that offer downloadable PDFs often end their links with .pdf or provide a download button on the page.

D. Adding a Timestamp to Filenames

To avoid overwriting files, you could add a timestamp to each filename:

TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
FILENAME="${TIMESTAMP}_$(basename "$URL").html"

This will give you unique filenames like 20241113_123456_article1.html, which are organized by date and time.

Step 4: Advanced Options with curl

For those who want even more control, here are a few useful curl options you can add to your script:

• Silent Mode (-s): Hides curl’s progress bar. Add it to curl -sL ... if you want a cleaner output.
• Retry on Failure (--retry): Retries the download in case of failure.

Example:

curl -sL --retry 3 "$URL" -o "$SAVE_DIR/$FILENAME"

This will attempt the download up to 3 times in case of network issues.

---

Step 5: Putting It All Together

Here’s the final version of the script with all the modifications:

#!/bin/bash

echo "Enter the directory to save articles (default is ~/Articles):"
read CUSTOM_SAVE_DIR
SAVE_DIR="${CUSTOM_SAVE_DIR:-$HOME/Articles}"
mkdir -p "$SAVE_DIR"

echo "Enter URLs to download (separate by spaces):"
read -a URLS

for URL in "${URLS[@]}"; do
  TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
  FILENAME="${TIMESTAMP}_$(basename "$URL").html"
  
  curl -sL --retry 3 "$URL" -o "$SAVE_DIR/$FILENAME"
  
  echo "Downloaded: $FILENAME"
done

Step 6: Running the Script

Run your script with:

./download_articles.sh

The script will prompt you for a save location and URLs, download the articles, and save them with a timestamped filename. You’re all set to read offline!

---

Wrapping Up

With just a few lines of Bash and curl, you’ve got a personalized article downloader! Try expanding this further by adding features like automatic updates, downloading images or other media, or even sending files to a Kindle.

Now you’re ready to enjoy your offline reading sessions, all powered by a simple script.

Play Magic 2014 on Galaxy Note 8

The Samsung Galaxy Note 8.0 tablet is the greatest most portable tablet made by man, in my opinion. However, it has one major software flaw. It still thinks it is the Galaxy Note 2. Whoever ported the software to the note 8 forgot to tell it that its no longer a phone.

For example, most websites will switch to the mobile version when you visit them using the Note 8. Some tablet-only apps won't show up on the Google Play store, including Magic 2014.

Here I'll explain how to easily and officially run Magic 2014 duels of the planeswalkers on Samsung Galaxy Note 8 or any relatively new Android phone or tablet. It requires a temporary root. This means that after installing the game you can unroot your device to regain the warranty. Make sure you have checked "unknown sources" and "USB debugging" in the settings.

This method does not wipe your memory. All your apps, files, and settings will be safe and remain untouched. However, backup your important files just in case.

Steps:

-Download and install the app Framaroot directly on the Note 8.
-Open Framaroot and root the device using the Aragon method.
-restart.
-Download and install Busybox from Google Play store.
-Download and install Market Helper directly on the Note 8.
-Open market helper and choose tablet, Asus Trandformer,  United States and AT&T. Then, click on Activate. (don't worry, this will reset when you reboot)
-Now search for magic 2014 on Google Play and install it.

If you still can't find the game on Google Play, visit play.google.com, log in with your Google account, and search for the game there. Then, install it on your device from the website. It will start downloading on your device immediately.

Now that the game is installed and runs fine, its time to unroot:

-Uninstall Busybox
-Uninstall Market Helper.
-Open Framaroot and choose Unroot. Then uninstall Framaroot.

That's it!

Magic 2014 runs smoothly on the Samsung Galaxy Note 8. The in-app purchases work fine. Rooting doesn't mess  with your device. And unrooting doesn't break the game. The world is a better place now.

Samsung Galaxy S4 Useless Features

Unlike most of the technology news reporters, I have enjoyed Samsung's Galaxy S4 launch event. I liked how they made fun of broadway musicals and laughed out loud when the Chinese guy who wasn't supposed to know English said "I'll be in my locker room if you need me." You can watch the full event in the video below

Having said that, the phone itself is awesome. The Galaxy S4 (not SIV) is the best phone on the planet right now. It has the best hardware features you can get in a mobile phone. The software features, however, are a different story.

It is a good idea to stuff a lot of software features in a device to market it. You can use the features you want and disable the ones you don't want. But what if you don't want all the new features of the S4? You turn them off. And thus you get the Galaxy S4 hardware in Galaxy S3 software.

I can list all the new software gimmicks and give each one a single reason that it will annoy the user, but that will take pages to write. I'll mention only a few:

- S-Pause: pausing a video when you're not looking at it is a great idea. But what if I wanna take a drink of water and don't want it to pause?
- S-Frame: having a picture of my face in the picture I'm taking is a great idea. But what if I don't want to include myself in the shot? By the time I disable it, the Kodak moment might pass.
- Air-flip: waving my hand in the air to go to the next photo. But what if I scratch my neck and it sees my elbow waving around, it will flip to the next photo.
And the list goes on and on.

All I'm saying is that all of these feature are good and useful but may get annoying with time. And the majority of users will never use the majority of features.

The only solution in my point of view is to implement a switch to toggle all these software features on and off. And make that switch easily accessible.

Otherwise I suggest that Samsung would focus its $400 million advertising campaign on the hardware features. Focus on the 5 inch screen and the 8-core CPU Not video pausing. I can't think of any situation where the 8-core CPU feature can be annoying!

Tomotiki.com & Kuwait Blogs Are Dead

Tomotiki.com was a blog aggregator. It had four seperate aggregators actually: Kuwait Blogs, Saudi Blogs, Cooking Blogs, and Ubuntu Linux Blogs.

Kuwait Blogs was the most popular site, generating over 4000 visits a day. For a trip down memory lane and to see the site again you can visit the Web Archive.

Tomotiki was forcefully closed by a takedown action. I could've contested the action and reopened it in less than a day (by mentioning that its an RSS reader) but I decided not to reopen it.

The reasons Tomotiki will remain closed are the following:

The main reason is the nudity and semi nudity which appears through the many blogs that are indexed, along with all the religious and political views that may or may not contradict with what I believe in.

Secondly, even though many bloggers email me daily to add their blogs to the website (it generates traffic to their blogs), I have had some complaints (and public threats and takedown actions), that I illegally steal other people's content. Obviously those people never heard of Google Reader or RSS.

The third reason is that the technology I use (which is basically WordPress with RSS plug-ins), is not suitable to handle as many blogs as I needed (I index of 100s of blogs).

That's why I have forwarded the domain to my blog.

I apologize if I didn't add your blog in time. I also apologize if you are used to read the blogs on Tomotiki and found that it disappeared. I hope you enjoyed it while it lasted.

Moving forward, I hope to find a solution for the main three points above. Namely, filter unwanted extreme posts, make sure all blogs are indexed with the approval/request of their owners, find a better platform to manage the aggregation process.

In the mean time I hope I can find a way to use google reader (or any web RSS reader) to index the list of blogs previously available on Tomotiki.

The lesson i learned from this project: The technology is ready but the people are not.

Playstation 4 Share Button

The share button on the PS4 is a really great idea. While playing, just hit the share button and your awesome move you just did will be available on youtube! This feature is not even available on the PC without external application that will slow down your computer. I'm looking forward to get my hand on a PS4 early next year.

Beautiful QR Codes

I went by a colorful booth in Civisama expo. They transform your ugly QR code to a beautiful colorful image. They can even include a logo in the middle of the QR code.  Check out their ads.

Piston Computer

I can stare at this picture all day. This is the Piston computer (or Steambox). I never expected a machine this small can have this many ports. This is perfect for everything, games, server, TV, browsing.. I hope it goes on sale soon.

Dell XPS 17 L702X Upgraded To Windows 8

I finally had the time to upgrade my humble gaming laptop. The upgrade from windows 7 to 8 took about 45 minutes and was very straight forward.

The only problem is that the touch screen stopped working. The N-trig drivers I downloaded from both Dell and N-trig website cannot run under Windows 8, even though it says Windows 8 on the download page. I googled around and came across a forum post recommending I deactivate Intel graphics. I did that and the touch screen worked after rebooting the machine.

Initial thoughts on the new Windows OS: it is fast, really fast. Arranging icons on the new start menus interface is not as responsive as arranging apps on iOS. I like the windows App Store as it has many good looking apps. unfortunately, most of which are website-apps, we got Apple to thank for that.

My Email To PC Gamer Magazine

Dear Future Publishing and PC Gamer,

Why don't you implement auto downloads of new digital magazines? Why do I have to keep the magazine app open for it to download? I have to keep touching the screen every few minutes to prevent the iPad screen from locking and therefore stopping the download process (ten minutes later)

Apple has given you the ability to auto download new issues on newsstand without opening the app. Please make use of it. Apple's example was "wake up in the morning to find your new issue waiting for you."

It usually takes me a few retries for every subscription to fully download the latest issue. It would be great if the new issue just shows up "waiting for me" every month.

Please implement auto download, I'm sure it's just a few lines of code to add this feature your platform. And I'm also sure it's going to make ALL your digital subscribers happy too.

If you might think that some subscribers might not like auto downloads, then make it optional in the app settings. It is a win-win situation.

I know this is a very long email but the issue is very important (no pun intended.)

Thanks,
Loyal Reader

Sent from my iPad

Updated Look at Kuwait Browser Market Share

Good news. Compared to the last time I checked, Internet Explorer usage is declining and Chrome usage is dramatically increasing. I'm guessing that by end of 2013 Chrome will finally be the leading browser in Kuwait.

Source: StatCounter Global Stats - Browser Market Share

Microsoft Surface Pro at Kuwait's Infoconnect

I went by the Microsoft booth at Infoconnect expo and got some hands-on time with the Surface pro. It is thin and light. Keyboard is very thin but thick enough to protect the screen. The Indian salesman at the Microsoft booth claimed the battery life is 6 to 8 hours. I believe it won't reach 5 hours in the best case.

The device will cost 199 Kuwait Dinars (around 700 US Dollars) with the keyboard. And it is already available (unofficially and with a higher price tag) at xcite and eureka stores all around Kuwait.

I also purchased a full version Windows 8 for 25 KD ($88) and was offered the Windows 8 pro upgrade for 19 KD but declined. I believe 6 KD difference is worth it to go for full version. The salesman also gave me anti-virus software for free (which I threw away almost immediately).

3 ways to transfer Playstation 3 save games

Im trying to move my saves to my new PS3 and found these easy methods:

Data Transfer Utility using an ethernet cable between the two PS3s to move everything from one PS3 to the other. This will pave over one PS3 with the data from another PS3. There is no merging of data.

Saved Data Utility lets you use a USB thumb drive to manually copy game saves to a USB drive and to other PS3s. The only drawback is that you can't copy game saves that are copy-protected. Not all game saves are copy-protected. Saved Data Utility also lets you use the PlayStation Network Online Storage that does let you copy copy-protected data. That requires a paid PlayStation Plus subscription.

Backup Utility will make a full system backup onto an external USB hard drive, but it will only restore copy-protected data onto the same console. If you restore onto a different console, everything but the copy-protected data will copy over

I think I'll manually copy my saves, hoping that none are copy-protected.

The iPad mini "mini review"

The ipad mini arrived. In kuwait you can get one from x-cite alghanim. They sell it online for 119 kd and in the store for 139. And they only sell the 16 GB wifi model.

The iPad mini is really thin and light. It feels good in the hand and is really snappy and fast. The screen is great except for the pixilation of really small text.

I recommend it for those of you who always carry your ipad with you. Its portable like a phone and usable like the 9.7 inch ipad.

Windows 8 pro is here

I just bought a hard copy of Windows 8 pro for $69 from Dubai international airport on my way back to kuwait. I'll try it out on a virtual machine in my office, if i like it I'll upgrade my office PC, if not, I'll upgrade my home PC.

I'll post my review here.

Oh and by the way, this blog is officially back. I'll post on a weekly basis, give or take a week or two..

Google I/O Keynote Day 2 Video

The video of keynote of Google io day 2, is here.