How to Use curl in a Bash Script to Download News Articles for Offline Viewing

Chat history

Ever wish you could save your favorite news articles to read offline, whether for an airplane ride or a cozy, no-WiFi spot? Using curl with a Bash script is a great way to download web content in seconds. This guide will show you how to create a script to fetch articles with curl, then add some options for those who want to personalize the process.

 

 

Why Use curl?

curl is a command-line tool for transferring data to and from servers. It’s fast, flexible, and widely supported, which makes it perfect for downloading articles, images, and more with a simple script.

---

Step 1: Setting Up the Script

First, let’s set up our script to download a list of articles. Create a file called download_articles.sh:

nano download_articles.sh

Copy this starter script into your file:

#!/bin/bash

# Directory to save downloaded articles
SAVE_DIR="$HOME/Articles"
mkdir -p "$SAVE_DIR"  # Create the directory if it doesn't exist

# List of article URLs to download
URLS=(
  "https://example.com/news/article1"
  "https://example.com/news/article2"
  "https://example.com/news/article3"
)

This code will save articles to an Articles folder in your home directory and sets up a list of example URLs to download.

---

Step 2: Download Articles Using curl

To download each article, we’ll loop through our list of URLs and use curl to fetch each one. We’ll use basename to give each file a meaningful name based on its URL:

for URL in "${URLS[@]}"; do
  # Extract a filename from the URL
  FILENAME=$(basename "$URL").html
  
  # Download the article with curl
  curl -L "$URL" -o "$SAVE_DIR/$FILENAME"
  
  echo "Downloaded: $FILENAME"
done

Here:

• basename extracts the last part of the URL (e.g., article1), and we add .html for readability.
• curl -L follows any redirects (handy for news sites).
• -o "$SAVE_DIR/$FILENAME" saves the file in our Articles folder with a unique name.

Run the script by making it executable and then executing it:

chmod +x download_articles.sh
./download_articles.sh

Step 3: Customizing the Script for Extra Flexibility

Want to give your script a personal touch? Here are some easy modifications to make it even more versatile.

A. Prompting for URLs

You can allow users to enter URLs each time they run the script:

echo "Enter URLs to download (separate by spaces):"
read -a URLS  # Take input as an array

Now, when the script runs, it’ll prompt the user to enter article URLs, which it will then use to download.

B. Setting a Custom Save Directory

To allow users to specify a custom directory:

echo "Enter the directory to save articles (default is $SAVE_DIR):"
read CUSTOM_SAVE_DIR
SAVE_DIR="${CUSTOM_SAVE_DIR:-$SAVE_DIR}"  # Use default if no input

mkdir -p "$SAVE_DIR"  # Ensure the directory exists

Now, the user can specify a directory to save articles, or press Enter to use the default Articles folder.

C. Downloading in Different Formats (PDF)

Some websites offer PDF versions of articles. If so, you can specify these URLs to download PDFs instead of HTML:

URLS=(
  "https://example.com/news/article1.pdf"
  "https://example.com/news/article2.pdf"
)

Tip: Many sites that offer downloadable PDFs often end their links with .pdf or provide a download button on the page.

D. Adding a Timestamp to Filenames

To avoid overwriting files, you could add a timestamp to each filename:

TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
FILENAME="${TIMESTAMP}_$(basename "$URL").html"

This will give you unique filenames like 20241113_123456_article1.html, which are organized by date and time.

Step 4: Advanced Options with curl

For those who want even more control, here are a few useful curl options you can add to your script:

• Silent Mode (-s): Hides curl’s progress bar. Add it to curl -sL ... if you want a cleaner output.
• Retry on Failure (--retry): Retries the download in case of failure.

Example:

curl -sL --retry 3 "$URL" -o "$SAVE_DIR/$FILENAME"

This will attempt the download up to 3 times in case of network issues.

---

Step 5: Putting It All Together

Here’s the final version of the script with all the modifications:

#!/bin/bash

echo "Enter the directory to save articles (default is ~/Articles):"
read CUSTOM_SAVE_DIR
SAVE_DIR="${CUSTOM_SAVE_DIR:-$HOME/Articles}"
mkdir -p "$SAVE_DIR"

echo "Enter URLs to download (separate by spaces):"
read -a URLS

for URL in "${URLS[@]}"; do
  TIMESTAMP=$(date +"%Y%m%d_%H%M%S")
  FILENAME="${TIMESTAMP}_$(basename "$URL").html"
  
  curl -sL --retry 3 "$URL" -o "$SAVE_DIR/$FILENAME"
  
  echo "Downloaded: $FILENAME"
done

Step 6: Running the Script

Run your script with:

./download_articles.sh

The script will prompt you for a save location and URLs, download the articles, and save them with a timestamped filename. You’re all set to read offline!

---

Wrapping Up

With just a few lines of Bash and curl, you’ve got a personalized article downloader! Try expanding this further by adding features like automatic updates, downloading images or other media, or even sending files to a Kindle.

Now you’re ready to enjoy your offline reading sessions, all powered by a simple script.

Play Magic 2014 on Galaxy Note 8

The Samsung Galaxy Note 8.0 tablet is the greatest most portable tablet made by man, in my opinion. However, it has one major software flaw. It still thinks it is the Galaxy Note 2. Whoever ported the software to the note 8 forgot to tell it that its no longer a phone.

For example, most websites will switch to the mobile version when you visit them using the Note 8. Some tablet-only apps won't show up on the Google Play store, including Magic 2014.

Here I'll explain how to easily and officially run Magic 2014 duels of the planeswalkers on Samsung Galaxy Note 8 or any relatively new Android phone or tablet. It requires a temporary root. This means that after installing the game you can unroot your device to regain the warranty. Make sure you have checked "unknown sources" and "USB debugging" in the settings.

This method does not wipe your memory. All your apps, files, and settings will be safe and remain untouched. However, backup your important files just in case.

Steps:

-Download and install the app Framaroot directly on the Note 8.
-Open Framaroot and root the device using the Aragon method.
-restart.
-Download and install Busybox from Google Play store.
-Download and install Market Helper directly on the Note 8.
-Open market helper and choose tablet, Asus Trandformer,  United States and AT&T. Then, click on Activate. (don't worry, this will reset when you reboot)
-Now search for magic 2014 on Google Play and install it.

If you still can't find the game on Google Play, visit play.google.com, log in with your Google account, and search for the game there. Then, install it on your device from the website. It will start downloading on your device immediately.

Now that the game is installed and runs fine, its time to unroot:

-Uninstall Busybox
-Uninstall Market Helper.
-Open Framaroot and choose Unroot. Then uninstall Framaroot.

That's it!

Magic 2014 runs smoothly on the Samsung Galaxy Note 8. The in-app purchases work fine. Rooting doesn't mess  with your device. And unrooting doesn't break the game. The world is a better place now.

Samsung Galaxy S4 Useless Features

Unlike most of the technology news reporters, I have enjoyed Samsung's Galaxy S4 launch event. I liked how they made fun of broadway musicals and laughed out loud when the Chinese guy who wasn't supposed to know English said "I'll be in my locker room if you need me." You can watch the full event in the video below

Having said that, the phone itself is awesome. The Galaxy S4 (not SIV) is the best phone on the planet right now. It has the best hardware features you can get in a mobile phone. The software features, however, are a different story.

It is a good idea to stuff a lot of software features in a device to market it. You can use the features you want and disable the ones you don't want. But what if you don't want all the new features of the S4? You turn them off. And thus you get the Galaxy S4 hardware in Galaxy S3 software.

I can list all the new software gimmicks and give each one a single reason that it will annoy the user, but that will take pages to write. I'll mention only a few:

- S-Pause: pausing a video when you're not looking at it is a great idea. But what if I wanna take a drink of water and don't want it to pause?
- S-Frame: having a picture of my face in the picture I'm taking is a great idea. But what if I don't want to include myself in the shot? By the time I disable it, the Kodak moment might pass.
- Air-flip: waving my hand in the air to go to the next photo. But what if I scratch my neck and it sees my elbow waving around, it will flip to the next photo.
And the list goes on and on.

All I'm saying is that all of these feature are good and useful but may get annoying with time. And the majority of users will never use the majority of features.

The only solution in my point of view is to implement a switch to toggle all these software features on and off. And make that switch easily accessible.

Otherwise I suggest that Samsung would focus its $400 million advertising campaign on the hardware features. Focus on the 5 inch screen and the 8-core CPU Not video pausing. I can't think of any situation where the 8-core CPU feature can be annoying!

Tomotiki.com & Kuwait Blogs Are Dead

Tomotiki.com was a blog aggregator. It had four seperate aggregators actually: Kuwait Blogs, Saudi Blogs, Cooking Blogs, and Ubuntu Linux Blogs.

Kuwait Blogs was the most popular site, generating over 4000 visits a day. For a trip down memory lane and to see the site again you can visit the Web Archive.

Tomotiki was forcefully closed by a takedown action. I could've contested the action and reopened it in less than a day (by mentioning that its an RSS reader) but I decided not to reopen it.

The reasons Tomotiki will remain closed are the following:

The main reason is the nudity and semi nudity which appears through the many blogs that are indexed, along with all the religious and political views that may or may not contradict with what I believe in.

Secondly, even though many bloggers email me daily to add their blogs to the website (it generates traffic to their blogs), I have had some complaints (and public threats and takedown actions), that I illegally steal other people's content. Obviously those people never heard of Google Reader or RSS.

The third reason is that the technology I use (which is basically WordPress with RSS plug-ins), is not suitable to handle as many blogs as I needed (I index of 100s of blogs).

That's why I have forwarded the domain to my blog.

I apologize if I didn't add your blog in time. I also apologize if you are used to read the blogs on Tomotiki and found that it disappeared. I hope you enjoyed it while it lasted.

Moving forward, I hope to find a solution for the main three points above. Namely, filter unwanted extreme posts, make sure all blogs are indexed with the approval/request of their owners, find a better platform to manage the aggregation process.

In the mean time I hope I can find a way to use google reader (or any web RSS reader) to index the list of blogs previously available on Tomotiki.

The lesson i learned from this project: The technology is ready but the people are not.

Playstation 4 Share Button

The share button on the PS4 is a really great idea. While playing, just hit the share button and your awesome move you just did will be available on youtube! This feature is not even available on the PC without external application that will slow down your computer. I'm looking forward to get my hand on a PS4 early next year.

Beautiful QR Codes

I went by a colorful booth in Civisama expo. They transform your ugly QR code to a beautiful colorful image. They can even include a logo in the middle of the QR code.  Check out their ads.

Piston Computer

I can stare at this picture all day. This is the Piston computer (or Steambox). I never expected a machine this small can have this many ports. This is perfect for everything, games, server, TV, browsing.. I hope it goes on sale soon.

Dell XPS 17 L702X Upgraded To Windows 8

I finally had the time to upgrade my humble gaming laptop. The upgrade from windows 7 to 8 took about 45 minutes and was very straight forward.

The only problem is that the touch screen stopped working. The N-trig drivers I downloaded from both Dell and N-trig website cannot run under Windows 8, even though it says Windows 8 on the download page. I googled around and came across a forum post recommending I deactivate Intel graphics. I did that and the touch screen worked after rebooting the machine.

Initial thoughts on the new Windows OS: it is fast, really fast. Arranging icons on the new start menus interface is not as responsive as arranging apps on iOS. I like the windows App Store as it has many good looking apps. unfortunately, most of which are website-apps, we got Apple to thank for that.

My Email To PC Gamer Magazine

Dear Future Publishing and PC Gamer,

Why don't you implement auto downloads of new digital magazines? Why do I have to keep the magazine app open for it to download? I have to keep touching the screen every few minutes to prevent the iPad screen from locking and therefore stopping the download process (ten minutes later)

Apple has given you the ability to auto download new issues on newsstand without opening the app. Please make use of it. Apple's example was "wake up in the morning to find your new issue waiting for you."

It usually takes me a few retries for every subscription to fully download the latest issue. It would be great if the new issue just shows up "waiting for me" every month.

Please implement auto download, I'm sure it's just a few lines of code to add this feature your platform. And I'm also sure it's going to make ALL your digital subscribers happy too.

If you might think that some subscribers might not like auto downloads, then make it optional in the app settings. It is a win-win situation.

I know this is a very long email but the issue is very important (no pun intended.)

Thanks,
Loyal Reader

Sent from my iPad

Updated Look at Kuwait Browser Market Share

Good news. Compared to the last time I checked, Internet Explorer usage is declining and Chrome usage is dramatically increasing. I'm guessing that by end of 2013 Chrome will finally be the leading browser in Kuwait.

Source: StatCounter Global Stats - Browser Market Share

Microsoft Surface Pro at Kuwait's Infoconnect

I went by the Microsoft booth at Infoconnect expo and got some hands-on time with the Surface pro. It is thin and light. Keyboard is very thin but thick enough to protect the screen. The Indian salesman at the Microsoft booth claimed the battery life is 6 to 8 hours. I believe it won't reach 5 hours in the best case.

The device will cost 199 Kuwait Dinars (around 700 US Dollars) with the keyboard. And it is already available (unofficially and with a higher price tag) at xcite and eureka stores all around Kuwait.

I also purchased a full version Windows 8 for 25 KD ($88) and was offered the Windows 8 pro upgrade for 19 KD but declined. I believe 6 KD difference is worth it to go for full version. The salesman also gave me anti-virus software for free (which I threw away almost immediately).

3 ways to transfer Playstation 3 save games

Im trying to move my saves to my new PS3 and found these easy methods:

Data Transfer Utility using an ethernet cable between the two PS3s to move everything from one PS3 to the other. This will pave over one PS3 with the data from another PS3. There is no merging of data.

Saved Data Utility lets you use a USB thumb drive to manually copy game saves to a USB drive and to other PS3s. The only drawback is that you can't copy game saves that are copy-protected. Not all game saves are copy-protected. Saved Data Utility also lets you use the PlayStation Network Online Storage that does let you copy copy-protected data. That requires a paid PlayStation Plus subscription.

Backup Utility will make a full system backup onto an external USB hard drive, but it will only restore copy-protected data onto the same console. If you restore onto a different console, everything but the copy-protected data will copy over

I think I'll manually copy my saves, hoping that none are copy-protected.

The iPad mini "mini review"

The ipad mini arrived. In kuwait you can get one from x-cite alghanim. They sell it online for 119 kd and in the store for 139. And they only sell the 16 GB wifi model.

The iPad mini is really thin and light. It feels good in the hand and is really snappy and fast. The screen is great except for the pixilation of really small text.

I recommend it for those of you who always carry your ipad with you. Its portable like a phone and usable like the 9.7 inch ipad.

Windows 8 pro is here

I just bought a hard copy of Windows 8 pro for $69 from Dubai international airport on my way back to kuwait. I'll try it out on a virtual machine in my office, if i like it I'll upgrade my office PC, if not, I'll upgrade my home PC.

I'll post my review here.

Oh and by the way, this blog is officially back. I'll post on a weekly basis, give or take a week or two..

Google I/O Keynote Day 2 Video

The video of keynote of Google io day 2, is here.

Download from two internet connections in one computer

With the recently-introduced download cap in Kuwait, I had to figure out a different method of downloading my files. Now I use my Viva 3G mifi card as a second internet line, in addition to the home DSL (at the same time).
I looked everywhere for a reasonably easy way to download from both sources without buying additional hardware or messing with complicates configuration files and/or scripts. I simply use a virtual machine and forward to it the entire 3G card connection.

On your Ubuntu (or windows) computer, set up a virtual Ubuntu machine with bare minimum software and select the 3G as the network source. Now share one of the computer folders with the virtual machine and start downloading on both connections simultaneously.
If you're having any problems with this method let me know in the comments section below this post.

New Web Sites

Everyone who reads my blog knows about my passion for automated websites and content aggregators. I've been busy lately throwing darts on a board to see what sticks out there. I created Tomotiki.com as my test chamber. And many projects were created on that domain. Of those tries and misses, 3 web sites have made it with enough daily page loads to break-even.

My new websites:

• Kuwait Blogs
• Cooking & Food Blogs
• Ubuntu News & Blogs

Feel free to spread the word and visit them according to your interests.

As always, I'm always listening for feedback and suggestions.

Kuwait Web Browser Market Share

It is with my deepest regret and pain that I announce the web browser market share in Kuwait.

Source: StatCounter Global Stats - Browser Market Share

As you can see, Internet Explorer is dominating Kuwait, a sign of complete computer illiteracy in my opinion. In comparison, IE is the second most popular web browser in Europe.

I am hugely disappointed but I will continue to spread the word about better web browsers, like Firefox, Chrome, and Safari.

Take care of your passwords

It is actually good to hear the latest breach of Gawker.com database and the release of millions of user information including passwords. It is always good to remind everyone to change their passwords, especially easy ones. And here I'll explain why it's not a good idea to have easy passwords and will give you some tips to help you create strong passwords.

To begin with, I'll explain how a server stores user passwords in a database. Let's say a user goes to hotmail.com and creates a new email with the password "password". The server will most likely encrypt the password and stores it in a database. Let's say the encrypted password is "£K%Ggh", so this strange word will be stored next to his email address. The next time the guy wants to check his email, he'll enter his username and password and the server will encrypt the password again and checks the encrypted word with the one stored in the database. If they match, the server will allow him access to his email.
Let's say the same user creates a commenting account in Gawker.com and uses the same email and password. Gawker will use another encryption scheme to generate their own encrypted word and store it in their own database. Let's say Gawker's word is "G?juu&".
Now if a hacker gets access to Gawker's database and encryption scheme, he can guess easy passwords like "password", "qwerty", "123456" and he'll see the encrypted word for each of them and then he'll search the database for users who have that word as a password and get their associated emails. He then uses the same password on their email accounts (hotmail in our example) and most likely the hacker will gain access to the user's email. And the same thing can happen to Twitter and Facebook accounts as well.

That's what happened today. And to add insult to injury, Gawker hackers released the whole database on the Internet. Allowing everyone access to the database.

In this particular case, if the password is hard to guess, the user will be better protected. So if the password was "pa$$word" or "Passw0rD" it will be a lot harder to guess. I'll give recommendations for better passwords later.

So, what you can do to protect your online account and what lessons we learned from today's event?

First, change all your passwords immediately  even if you are not registered in Gawker or it's sister websites. Second, don't use the same password for different accounts. Third, use special characters in your passwords (other than letters and numbers). Forth, use an app on your mobile phone to store your passwords in case you forget them.

Finally, how to choose a good password:
-Add a mix of these symbols: ¥$€><~|\_.,?!'&£/-@
-Mix upper and lower case letters with numbers
-Replace letters with similar symbols: "password" can be "P@$$w0Rd"

-Intentionally misspell words
-Use a mix of birthdays, occasions, acrynoms, site names, symbols: "20!12!1978" "&face~book&"

Just be creative with your password to prevent anyone from gaining access to your personal information. And good luck out there.

The State of Twitter in Kuwait

It is amazin how Twitter adoption in Kuwait is accelerating. It was about three years ago when i last searched for tweets near Kuwait. I found less than 100 tweets spanning 24 hours. Now, everyone and their mom is tweeting. When you search for tweets near Kuwait you'll find 100 tweets every 5 minutes and more at times.

The good thing about twitter, in my opinion, is that it's not restricted for one thing. It is there for you to use it how you please. And there is absolutely no wrong way to use twitter.

Personally, I use twitter to write about interesting stuff that I read online, report news that are important to me, talk about funny things I come across, and chat with friends. That's my contribution to twitter. However, I consume more than I contribute. I follow all sorts of technology celebrities and Kuwaitis who have the same interests as me, mainly technology and games. I also follow my friends. And use Flipboard to see the links posted by the people I'm following.
But this post is about my observation of Kuwaiti tweeps (people who use twitter) and their behavior. I think they are divided into three categories. The ones who use it to chat only, influenced by blackberry maybe? The ones who use it to post jokes and funny pictures, blackberry influence too? And the ones who have no sense of privacy and broadcast every single thing they are doing, basically an open invitation to stalkers. And any combination of these three categories. And the confused newbies.

Again, this is twitter and there is no wrong way to use it. Just be careful and know that all your tweets are public and bad people may read your tweets and know that you are "at the avenues mall pinkberry at the table near the door and just went to the bathroom alone." Or you "left the Porsche doors open at 360 parking level 2 ." Just don't do that, people.

I'm constantly monitoring the adoption of new technology in Kuwait and will try to take another look at twitter in Kuwait in about 3 years from now and see what changes.

Happy twittering.